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1.  INTRODUCTION. 


1.1  GENERAL. 


The  Screener  Proficiency  Evaluation  and  Reporting  System  (SPEARS)  is  being  developed  to 
improve  and  maintain  the  effectiveness  of  security  screening  personnel  employed  at  airport 
security  checkpoints.  The  SPEARS  consists  of  two  components;  (a)  an  off-line  Computer-Based 
Training  (CBT)  system  to  teach  screeners  to  detect  various  threat  objects,  and  (b)  an  on-line 
threat  image  projection  (TIP)  training  and  testing  component  employed  at  airport  security 
checkpoints.  This  latter  configuration  is  to  further  develop  and  maintain  threat  detection 
proficiency  by  projecting  simulated  X-ray  images  of  threat  objects  into  X-ray  images  of  actual 
passenger  bags.  The  effectiveness  of  the  CBT  and  TIP  components  will  be  addressed  during 
separate  test  and  evaluation  (T&E)  activities. 

This  Test  and  Evaluation  Plan  (TEP)  addresses  the  Critical  Operational  Issues  and  Criteria 
(COICs)  and  Critical  Technical  Issues  and  Criteria  (CTICs)  established  by  the  Federal  Aviation 
Administration  (FAA)  for  the  TIP  component  of  the  SPEARS. 

1.2  PURPOSE. 


The  SPEARS  Operational  Test  and  Evaluation  (OT&E)  for  the  TIP  component  is  being 
conducted  to  evaluate  the  ability  of  the  SPEARS  to  improve,  maintain,  and  monitor  airport  X-ray 
screener  performance.  Maintaining  a  workforce  of  adequately  trained  and  performing  X-ray 
screening  personnel  is  critical  to  the  mission  of  aviation  security,  both  domestically  and 
internationally.  This  TEP  outlines  the  methods  and  procedures  to  be  used  in  ensuring  that  the 
candidate  SPEARS  devices  meet  the  functional  requirements  established  by  the  FAA  as  being 
necessary  to  produce  this  workforce.  These  methods  include  the  collection  and  analysis  of 
empirical  data  on  the  operational  capabilities  of  candidate  SPEARS  devices. 

1.3  SCOPE. 


The  TEP  will  be  executed  during  an  OT&E  to  evaluate  the  EG&G  Astrophysics  Linescan 
Training  and  Testing  System  (TnT™)  on-line  training  and  testing  component.  The  TnT™  is 
currently  the  only  SPEARS  TIP  device  whose  manufacture  has  agreed  to  an  FAA  evaluation. 
The  OT&E  will  be  conducted  in  two  parts  and  the  COICs  will  be  evaluated  during  operational 
testing  at  the  Los  Angeles  International  Airport  (LAX).  Following  operational  testing,  the  CTICs 
will  be  evaluated  during  technical  testing,  either  in  the  FAA  Aviation  Security  Laboratory  or  at 
the  vendor  facility.  The  T&E  will  focus  on  determining  how  effective  the  SPEARS  is  in  meeting 
the  functional  requirements  embodied  in  both  the  COICs  and  CTICs,  and  in  complying  with 
Human  Factors  Engineering  (HFE)  design  principles. 

SPEARS  provides  training  for  many  categories  of  threats.  Historically,  threat  objects  such  as 
weapons  have  resulted  in  the  highest  detection  rates  in  operational  settings.  In  many  cases,  the 
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detection  rates  associated  with  these  types  of  threats  would  be  difficult  to  improve. 
Comparatively,  detection  rates  for  Improvised  Explosive  Devices  (lEDs)  have  been  lower.  lED 
detection  performance  is,  therefore,  of  specific  concern  to  this  study,  and  operational  testing  will 
focus  on  the  ability  of  SPEARS  TIP  training  and  testing  to  improve  BED  detection  performance. 
Specifically,  lED  detection  performance  for  one  group  of  subjects  trained  using  the  SPEARS  TIP 
component  will  be  compared  with  a  second  (control)  group  of  subjects  not  receiving  such 
training.  To  ensure  that  the  T&E  addresses  operational  effectiveness,  subjects  will  consist  of 
certified  airport  screeners  carrying  out  their  normal  screening  duties  at  LAX. 

1.4  BACKGROUND. 


1 .4. 1  SPEARS  Program  Background. 


The  SPEARS  Program  was  initiated  in  response  to  a  congressional  mandate  (Aviation  Security 
Improvement  Act  of  1990,  Public  Law  101-604)  tasking  the  FAA  to  improve  aviation  security 
through  the  optimization  of  Human  Factors  (HF)  elements  in  the  U.S.  airport  security  system. 
The  issue  of  screener  performance  and  effectiveness  was  highlighted  as  an  area  requiring 
evaluation  to  identify  potential  security  improvements.  An  aviation  security  Department  of 
Transportation  (DOT)  task  force  supported  this  thrust  by  concluding  that  human  performance 
was  the  critical  element  in  the  screening  process. 

The  mandate  directed  that  screeners  be  effectively  trained  to  use  threat  detection  equipment 
properly.  The  detection  of  explosive  and  incendiary  devices  was  critically  important  because  of 
the  potential  for  significant  damage,  causing  a  crucial  loss  of  life  and  aviation  resources. 

The  FAA’s  Aviation  Security  Human  Factors  (AvSec  HF)  Program  published  initial  SPEARS 
functional  descriptions  in  the  Commerce  Business  Daily  (CBD)  to  invite  vendors  to  develop 
candidate  SPEARS  devices.  Another  objective  of  the  CBD  announcement  was  to  determine  if 
any  available  commercial  equipment  existed  that  could  potentially  meet  the  SPEARS  objectives 
and  functional  requirements. 

This  research  is  sponsored  by  the  AvSec  HF  Program,  Office  of  Aviation  Security  Research  and 
Development  (AAR-510),  located  at  the  FAA  Technical  Center  in  Atlantic  City,  New  Jersey. 
Support  for  the  research  is  provided  by  the  FAA  Simulation  and  Human  Factors  Division  (ACT- 
500),  also  located  at  the  FAA  Technical  Center,  and  by  AAR-510  support  contractors. 

1 .4.2  X-Rav  Screener  Duties. 


Airport  security  checkpoints  are  located  outside  of  airline  boarding  areas  and  typically  include  a 
moving  conveyor  belt,  an  X-ray  screening  device  and  an  X-ray  display  monitor.  Passenger 
baggage  is  carried  by  the  conveyor  belt  through  the  X-ray  equipment,  which  produces  images  of 
the  bag  contents.  The  X-ray  images  are  then  displayed  on  the  monitor  for  evaluation  by  certified 
screeners. 
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X-ray  screeners  generally  work  8-hour  shifts,  comprising  15-  to  45-niinute  shift  intervals 
separated  by  5-  to  10-minute  breaks.  Screeners  work  in  teams  of  two  or  more;  however,  only  one 
screener  is  assigned  to  observe  a  given  X-ray  monitor  at  any  one  time.  After  baggage  is  loaded 
onto  the  conveyor  belt  and  scanned,  the  X-ray  image  is  presented  to  the  screener  observing  the 
monitor.  The  screener  visually  scans  the  image  and  performs  an  initial  image  analysis.  Image 
features  are  analyzed  for  attributes  such  as  size,  profile,  angle,  shape,  and  content  components. 
The  images  may  be  enhanced  by  selected  screen  functions  such  as  color  coding,  comparing 
black/white  and  color  images,  and  magnifying  sections  of  the  image. 

After  initial  image  analysis  and  enhancement,  the  screener  must  decide  whether  an  obvious  or 
potential  threat  exists.  The  decision  can  result  in  one  of  the  following  five  outcomes. 

a.  No  Threat  -  If  all  items  can  be  identified  and  the  screener  determines  that  no  threat 
exists,  the  bag  is  passed  on  and  the  screener  begins  image  analysis  of  the  next  bag. 

b.  Unidentified  Items  -  If  all  items  cannot  be  positively  identified,  the  screener  may 
reposition  the  bag  and  rescan.  If  the  items  are  still  not  identifiable,  the  screener  will  call  for  a  bag 
check  (usually  a  hand  check  performed  by  another  security  worker)  and  continue  scanning  duties. 

c.  Potential  Threat  -  If  a  potential  threat  is  detected  or  objects  cannot  be  identified 
due  to  clutter,  opaque  shading,  or  positioning;  or  if  suspicious  electronic  devices  are  detected;  the 
screener  may  stop  the  belt,  alternate  the  position  of  the  baggage,  and  rescan.  If  the  baggage 
objects  are  still  unidentifiable,  or  are  in  any  way  suspicious,  the  screener  will  call  for  a  bag  check. 

d.  Nonpermissible  Item(s)  -  Nonpermissible  items  include  hazardous  or  toxic 
materials  like  mace,  chemicals,  gasoline,  and  flammable  liquids.  If  a  nonpermissible  item  is 
detected,  the  screener  will  call  for  a  bag  check  and  indicate  the  location  of  the  item(s)  detected  to 
the  security  team  member  performing  the  bag  check.  The  bag  is  then  released  to  the  team 
member  and  the  screener  will  continue  with  the  next  bag. 

e.  Obvious  Threat  -  If  the  screener  detects  an  obvious  threat  (such  as  a  gun,  knife, 
explosive,  or  grenade),  the  belt  is  stopped  immediately  and  the  article  is  retained  imder  the  X-ray 
scan.  The  screener  will  then  depress  an  emergency  button,  notify  the  supervisor  on  duty,  and 
instrua  the  baggage  owner  to  wait  until  the  baggage  contents  have  been  examined  and  secured. 
When  a  law  enforcement  agent  arrives,  the  screener  will  identify  the  area  and  item  of  concern. 
When  instructed,  the  screener  will  release  the  article  and  restart  the  belt. 

1.5  FUNCTIONAL  REQUIREMENTS. 


The  SPEARS  Tff  is  intended  to  provide  X-ray  screeners  and  supervisory  personnel  with 
recurrent  training,  performance  monitoring,  and  evaluation  capabilities.  A  successful  SPEARS 
should  ultimately  improve  the  airport  baggage  screener’ s  threat  detection  performance  in  two 
ways.  Exposure  to  a  large  image  library  of  threat  objects  should  enhance  screener  familiarity  with 
images  of  actual  threat  objects.  Screener  performance  monitoring  and  feedback,  accompanying 
false  image  insertion,  should  also  provide  increased  motivation  and  vigilance. 

This  TEP  addresses  only  the  requirements  for  the  on-line  testing  and  training  component.  The 
off-line  instructional  training  portions  of  the  SPEARS  were  investigated  in  an  earlier  T&E  (see 
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the  Test  and  Evaluation  Report  [TER]  for  the  Screener  Proficiency  Evaluation  and  Reporting 
System  [SPEARS]  Computer-Based  Training  [Fobes,  et  al.,  1995]).  Although  separated  for  the 
purposes  of  T&E,  the  SPEARS  components  should  be  used  in  conjunction  with  one  another  for 
optimal  training  of  screening  personnel. 

1.5.1  Literature  Review. 


For  the  SPEARS  to  be  effective,  it  must  increase  human  ability  in  the  identification  of  targets, 
namely  threat  objects.  The  aim  of  the  SPEARS  TIP  component  is  to  enhance  the  ability  of 
screeners  to  detect  threats  within  baggage  through  regular  exposure  to  a  variety  of  threat  images. 
This  exposure  is  accomplished  by  using  TIP  to  insert  FTIs  onto  the  X-ray  images  of  normal 
baggage  flow  being  X-rayed,  or  to  insert  CTIs  consisting  of  fictitious  images  of  passenger  bags 
containing  threat  images  into  the  normal  passenger  bag  image  flow.  TIP  exposure  is  expected  to 
enhance  detection  performance  in  the  following  two  ways. 

a.  Screeners  will  gain  mcperience  and  thereby  increase  their  ability  to  locate  and 
identify  a  diverse  array  of  threat  objects  under  normal  working  conditions. 

b.  Regular  exposure  to  threat  images  will  increase  screener  attention  levels,  thereby 
maintaining  the  screener’s  ability  to  detect  threats  in  cluttered  bags. 

The  expected  performance  effects  fi'om  the  SPEARS  TIP  component  are  supported  by  a  review 
of  the  applicable  HF  and  psychology  literature,  specifically  in  the  areas  of  Signal  Detection 
Theory  (SDT),  vigilance  theory,  and  training. 

1 . 5 . 1 . 1  Implications  of  Signal  Detection  Theory  for  the  SPEARS . 


SDT  considers,  in  part,  the  role  of  human  vigilance  in  human  detection  of  infi'cqucnt  target 
signals.  A  description  of  SDT  is  provided  in  Appendk  A.  Key  to  the  purpose  of  the  SPEARS 
TP  is  the  concept  of  vigilance  decrement.  The  vigilance  decrement  has  been  defined  as  a 
decrease  in  human  performance  as  a  function  of  time  on  task  (Wickens,  1992).  In  experimental 
paradigms,  signal  detection  rates  and  signal  detection  measures  (d'  [derived  operator  sensitivity] 
and  P  or  c  [Operator  Response  Criterion])  are  collected  in  time  increments,  with  the  results 
demonstrating  decreasing  signal  detection  performance  over  time.  Although  many  factors  can 
contribute  to  this  decrement,  the  factor  most  pertinent  to  X-ray  screening  of  airline  passenger 
baggage  is  criterion  shifts  associated  with  lower  levels  of  target  expectancy.  Because  the 
frequency  of  lED  presentation  for  the  screener  in  a  real  operational  setting  is  extremely  low,  the 
screener  may  be  unprepared  to  detect  a  threat  object  on  the  rare  occasion  when  one  is  presented. 

In  SDT  terms,  the  expectancy-derived  vigilance  decrement  is  attributed  to  an  upward  adjustment 
of  the  operator  response  criterion  in  response  to  a  reduction  in  the  perceived  frequency  (and 
therefore  expectancy)  of  target  events  (Wickens,  1992).  If  the  operator  response  criterion  is 
based  on  perceived  target  frequency  levels,  then  additional  missed  targets  may  result  in  the 
perception  of  even  lower  target  frequencies.  The  corresponding  effect  may  be  a  continual 
decrease  in  detection  performance. 
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Nadler,  Mengert,  and  Carpenter- Smith  (1994)  indicate  that  inserting  TIPs  into  the  normal  flow  of 
baggage  should  increase  the  expectancy  levels  of  the  baggage  screeners,  thereby  counteracting 
this  adverse  criterion  shift  and  increasing  overall  detection  rates  of  actual  threat  objects.  To 
increase  screener  expectancy  levels,  adequate  and  immediate  knowledge  of  results  for  both  real 
and  artificial  threats  is  required.  TIP  must  also  increase  operator  sensitivity  so  that  the  shift  in 
criterion  is  not  accompanied  by  an  increase  in  false  alarms. 

Operational  screener  watch  durations  are  relatively  short.  For  carry-on  (CO)  baggage  screeners, 
the  watch  duration  ranges  from  20  to  30  minutes,  depending  on  the  security  procedures  in  use  at 
the  particular  airport.  Similarly,  the  watch  durations  for  checked  baggage  screeners  rarely  exceed 
1  hour.  The  majority  of  vigilance  decrement  research  has  focused  on  tasks  with  watch  durations 
of  1  hour  or  greater,  usually  focusing  on  conventional  work  shifts  of  2-,  4-,  or  8-hour  watches. 
While  it  is  doubtful  that  the  degree  of  vigilance  decrements  detected  towards  the  end  of  the  eighth 
watch  hour  are  directly  applicable  to  a  20-minute  or  1  hour  session,  vigilance  decrement  effects 
are  expected  to  accumulate  across  multiple  screening  shifts. 

The  lack  of  actual  threat  presentation  over  several  weeks,  months,  or  years  is  expected  to  result  in 
dramatically  reduced  signal  expectancy.  This  effect  might  be  counteracted  through  the  systematic 
use  of  techniques  (e.g.,  by  regular  insertion  of  FTIs  and  CTIs  into  the  baggage  screening 
environment)  that  change  the  expectancy  of  threat  events.  According  to  SDT,  therefore,  the 
SPEARS  TIP  may  result  in  enhanced  screener  detection  performance  by  maintaining  appropriate 
response  criteria  over  extended  periods. 

1 .5 . 1 .2  Implications  of  Training  Theory  on  the  SPEARS. 


The  training  literature  reveals  implications  for  the  design  and  implementation  of  a  SPEARS 
device.  In  a  review  of  training  literature,  Goldstein  (1986)  found  that  distributed  practice  for 
procedural  skills,  such  as  X-ray  screening,  provides  the  most  advantageous  results  over  time. 
Massed  practice  sessions  tend  to  show  better  immediate  training  results  and  also  require  less 
overall  training  time  to  achieve  a  minimum  criteria.  Massed,  off-line,  instructional  training  is, 
therefore,  a  necessary  component  of  screener  training.  For  retention  over  extended  periods, 
however,  the  distributed  or  spaced  training  sessions  will  result  in  better  overall  performance.  TIP, 
as  a  regular  component  of  the  screener’ s  daily  activities,  should  result  in  improved  lED  detection 
performance. 

Practice  alone  will  not  provide  adequate  training  retention.  Adequate  retention  requires  a  series 
of  interacting  training  variables,  which  includes  an  adequate  knowledge  of  results  and  proper 
motivation  of  the  trainees.  Adequate  and  immediate  feedback  to  TIP  responses  (correct  and 
incorrect)  should  also  help  to  improve  screener  performance. 

Another  factor  critical  to  the  acquisition  and  subsequent  retention  of  job  skills  is  the  extent  to 
which  training  is  transferred  to  the  operational  environment.  The  TIP  concept  employs  the 
principle  of  identical  elements,  in  that  the  training  task  uses  the  identical  environment  and  stimuli 
to  that  of  the  actual  screening  task.  Training  theory  predicts  that  this  training  principle  should 
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result  in  a  high  level  of  transfer  of  TIP  training  to  the  operational  environment,  which  should 
result  in  increased  lED  detection  performance. 

1.6  SYSTEM  DESCRIPTION. 


For  this  SPEARS  TIP  OT&E,  only  one  vendor,  EG&G  Astrophysics  TnT™,  agreed  to  have  then- 
system  evaluated. 

1.61  EG&G  Astrophysics  TnT™. 


The  following  is  a  technical  description  of  the  TnT™  components  to  be  tested. 

a.  A  color  monitor  and  a  black/white  monitor  mounted  side-by-side,  just  as  the 
monitors  are  mounted  on  the  EG&G  Astrophysics  E-Scan  X-ray  machine. 

b.  A  control  panel  mounted  below  the  monitors  that  replicates  the  control  panel  on 
the  E-Scan  X-ray  machine. 

c.  A  trackball  located  to  the  right  of  the  control  panel  that  operators  use  to  interact 
with  the  training  programs. 

d.  A  computer  located  inside  the  equipment  housing  that  runs  the  training  programs. 

e.  A  variety  of  cables  and  connectors  used  to  coimect  the  training  system  to  the 
EG&G  Linescan  X-ray  and  to  the  recording  equipment,  such  as  video  recorders. 

1.6.2  X-Rav  Machine  Modifications. 


To  allow  for  the  connection  of  TnT™  workstations,  input  boards  will  be  installed  by  EG&G 
Astrophysics  to  two  of  the  Linescan  System  Ten  X-ray  machines  at  LAX,  Terminal  2. 

1.7  TEST  OVERVIEW. 


There  are  two  categories  of  issues  that  will  be  used  to  assess  the  candidate  SPEARS  against  the 
functional  requirements:  (a)  operational  issues  and  (b)  technical  issues.  As  these  issues  involve 
substantially  different  requirements  and  investigation  methods,  the  overall  SPEARS  FBP  OT&E  is 
similarly  partitioned. 

1.71  Critical  Operational  Issues  and  Criteria  Evaluation. 


The  Critical  Operational  Issues  and  Criteria  (COICs)  are  those  issues  and  criteria  necessary  to 
evaluate  the  SPEARS  operational  requirements  at  a  field  test  at  LAX.  The  field  test  will  involve 
collecting  data  to  assess  the  effectiveness  of  the  SPEARS  TIP  testing  and  training  on  screener 
lED  detection  performance.  Subjective  rating  and  comment  data  will  also  be  collected  to 
determine  the  level  of  usability  of  the  SPEARS  and  the  representativeness  of  threat  insertion  using 
Tff. 
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1 .12  Critical  Technical  Issues  and  Criteria  Evaluation. 


The  Critical  Technical  Issues  and  Criteria  (CTICs)  are  those  issues  and  criteria  necessary  to 
evaluate  the  SPEARS  against  technical  requirements,  such  as  system  customization,  screener 
capabilities  reporting,  interoperability,  feedback,  security,  insertion,  and  image  content.  The 
CTICs  will  be  investigated  using  structured  protocols  during  technical  testing  of  system  features. 
Examples  of  the  features  to  be  tested  are  the  number  of  possible  stored  images,  reporting 
capabilities,  and  the  ability  to  receive  new  threat  images  through  a  modem. 

1 .7.3  Test  Limitations  and  Impact. 


In  addition  to  measuring  the  vigilance  aspects  of  detecting  the  FTIs  and  CTIs,  the  training  aspects 
will  be  examined  using  a  computer-based,  black  and  white  X-ray  image  presentation  device.  This 
study  of  the  generalized  performance  improvements  will  involve  presenting  25  CO  test  bag 
images,  each  containing  a  Modular  Bomb  Set  (MBS),  randomly  inserted  into  a  series  of  175 
comparison  bag  images  (i.e.,  bags  without  MBSs).  The  primary  reason  for  using  the 
computerized  device  rather  than  actual  baggage  containing  MBSs  is  to  avoid  the  negative  effects 
that  multiple  trials  using  actual  bags  would  have  on  passenger  flow  and  airline  operations.  The 
computerized  testing  lacks  some  aspects  of  operational  representativeness,  but  the  effects  of  this 
diminished  representativeness  include  obtaining  the  screener’ s  maximal  capability.  The  screeners 
will  analyze  actual  X-ray  images  and  will  be  required  to  respond  quickly  to  each  image,  simulating 
the  passenger  flow  requirements  at  the  checkpoint. 

The  SPEARS  TIP  is  meant  to  be  tailored  by  supervisors  to  individual  screener  abilities,  training 
needs,  and  workload.  Such  tailored  training  and  testing  will  not  be  possible  during  this  OT&E, 
and  a  more  generic  training  and  testing  protocol  will  be  utilized  (see  section  2.4.2. 1).  Therefore, 
the  optimal  impact  of  the  SPEARS  TIP  training  may  not  be  achieved  for  some  of  the  screeners  in 
this  study. 

1  8  CRITICAL  OPERATIONAL  ISSUES  AND  CRITERIA. 


In  the  sections  that  follow,  the  top-level  operational  testing  procedures  will  be  discussed.  Each 
operational  issue  is  identified  along  with  its  criteria,  Measures  of  Performance  (MOPs),  and  test 
strategy. 

The  issues  will  be  investigated  through  screener  performance  data  collected  during  the  study,  and 
through  subjective  comments  and  ratings  collected  after  the  study’s  completion.  Additional  data 
for  the  usability  issue  will  be  collected  by  using  a  tailored  Usability  HFE  Checklist  (Appendk  B) 
based  on  the  Guidelines  for  the  Design  of  User  Interface  Software  (Smith  and  Mosier,  1986). 
The  primary  experimental  design  and  test  procedures  are  provided  in  sections  2.4  and  2.7,  which 
include  discussions  about  specialized  data  collection  needs  for  usability  and  insertion. 
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1.8.1  Training. 


1.8. 1.1  Issue. 


Does  TIP  enhance  screener  performance? 

Criterion  A.  Projecting  fictional  images  is  accompanied  by  a  sustained  increase  (statistical)  in  the 
screener’ s  probability  of  detection  (Pa)  of  DEDs. 

MOP  1.  The  Number  of  Hits  (Nh)  for  test  bag  images  collected  before  and  after  exposure 
to  the  SPEARS  treatment. 

NOTE:  The  relationship  between  Nh  and  Pa  will  be  discussed  in  section  2.7. 1 .2. 

Criterion  B.  Projecting  fictional  images  is  not  accompanied  by  a  sustained  increase  (statistical)  in 
the  screener’s  probability  of  false  alarm  (?&)  when  scanning  for  lEDs. 

MOP  2.  The  Number  of  False  Alarms  (Na)  for  comparison  bag  images  collected  before 
and  after  exposure  to  the  SPEARS  treatment. 

NOTE.  The  relationship  between  Nf,  and  Pfa  will  be  discussed  in  section  2.7. 1 .2. 

Criterion  C.  Projecting  fictional  images  is  accompanied  by  a  sustained  increase  (statistical)  in 
the  screener’s  d'. 

MOP  3.  The  d'  to  detect  the  test  bag  images  collected  before  and  after  exposure  to  the 
SPEARS  treatment. 

1.8. 1.2  Test  Strategy. 


MOP  data  will  be  collected  concurrently  for  two  groups  of  screeners  during  a  field  study  at  LAX. 
One  group  of  screeners,  the  experimental  group,  will  be  exposed  to  the  SPEARS  TIP.  The  other, 
the  control  group,  will  be  tested  without  exposure  to  the  SPEARS  TIP.  To  determine  the  effect 
of  SPEARS  training  on  lED  detection  performance,  Nh  and  Na  will  be  observed  and  recorded  for 
individual  screeners  three  times  during  the  study:  (a)  during  a  pretest  before  the  SPEARS  TIP 
training  begins  for  the  experimental  group,  (b)  during  a  posttest  4  weeks  after  the  pretest,  and  (c) 
during  a  second  posttest  8  weeks  after  the  first  posttest. 

The  MOP  data  will  be  used  to  derive  Pa  and  Pf,  for  each  screener.  The  values  of  Pa  and  P^  will  be 
compared  between  training  and  control  conditions  and  across  measurement  periods  to  determine 
the  effect  of  the  SPEARS  TIP  on  LED  detection  performance. 
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1.8.2  Vigilance. 


1.8.2. 1  Issue. 


Does  SPEARS  TIP  increase  and  sustain  screener  vigilance  in  detecting  threat  images? 

Criterion  A.  There  is  a  sustained  increase  (statistical)  in  Pa  of  FTIs  for  the  TIP  training  group 
during  the  first  4  weeks  of  the  study. 

MOP  5.  The  Nh  for  FTIs  collected  across  the  period  of  the  SPEARS  treatment. 

Criterion  B.  There  is  a  sustained  increase  (statistical)  in  Pa  of  FTIs  for  the  TIP  training  group 
during  the  last  8  weeks  of  the  study. 

MOP  6.  The  Nh  for  FTIs  collected  across  the  period  of  the  SPEARS  treatment. 

1. 8.2.2  Test  Strategy. 


MOP  data  will  be  collected  during  a  field  study  at  LAX.  The  SPEARS  device  will  be  installed  at 
the  airport  and  used  by  the  experimental  group  of  screeners  over  an  extended  period.  To 
determine  the  effect  of  SPEARS  TIP  on  screener  vigilance,  the  number  of  FTI  presentations  and 
Nh  will  be  observed  and  recorded  for  the  experimental  group  screeners  throughout  the  SPEARS 
intervention.  Unfortunately,  the  SPEARS  device  does  not  record  false  alarms  or  the  total  number 
of  passenger  bags  screened,  so  other  measures  (e.g.,  SDT  measures)  cannot  be  computed.  The 
resulting  Pd  measures  will  be  compared  across  weekly  segments  of  the  study  to  determine  if 
screener  performance  increases  fi'om  initial  levels  and  is  then  maintained  over  time. 

1.8.3  Usability. 


1.8.3. 1  Issue. 


Are  there  any  software  or  hardware  factors  or  procedural  aspects  that  degrade  the  screeners’  or 
supervisors’  ability  to  use  the  system  effectively? 

Criterion  A.  Investigative  in  nature. 

MOP  7.  Subjective  ratings  of  system  usability  by  screeners  and  supervisors. 

MOP  8.  HFE  Checklist  results. 
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1 . 8,3 .2  Test  Strategy. 


During  the  screener  off-line  performance  data  collection  sessions,  usability  evaluations  will  be 
performed.  The  usability  issue  is  relevant  to  the  supervisors  who  program  the  machine  and  to  the 
screeners  who  must  detect  and  respond  to  the  TIP  capability  (i.e.,  TIP  button  on  the  TnT™).  The 
usability  evaluations  will  consist  of  structured  interviews  (Appendix  C),  the  completion  of  rating 
scales  by  the  participating  screeners  and  supervisors,  and  the  application  of  a  t^ored  SPEARS 
Usability  Human  Factors  Engineering  (HFE)  checklist  (see  Appendix  B),  which  is  based  on  the 
Guidelines  for  the  Design  of  User  Interface  Software  (Smith  and  Mosier,  1986)  and  MEL-STD- 
1472D. 

1.9  CRITICAL  TECHNICAL  ISSUES  AND  CRITERIA 


This  section  identifies  each  technical  issue,  its  related  criteria,  MOPs,  and  test  strategy  to  be 
addressed  by  OT&E.  The  test  methods  to  be  used  include  the  following: 

a.  Analysis  -  A  method  of  verification  that  consists  of  comparing  hardware  or 
software  design  with  known  scientific  and  technical  principles,  technical  data,  procedures,  and 
practices  to  validate  that  the  proposed  design  will  meet  the  specified  functional  or  performance 
requirements. 

b.  Inspection  -  A  method  of  verification  to  determine  compliance  with  specification 
requirements  that  consists  primarily  of  visual  observations  or  mechanical  measurements  of 
equipment,  physical  locations,  or  technical  examination  of  engineering  support  documentation. 

c.  Study  -  Not  defined  in  DOT  Order  1810.4F  (1992),  but  a  study  usually  entails  a 
systematic  field  or  laboratory  investigation  and  is  the  ultimate  measure  of  the  usability  of  a  system. 
It  includes  the  verification  methods  of  analysis,  demonstration,  inspection,  and  test  (Cook  and 
Campbell,  1979). 

Several  of  the  technical  issues  for  this  OT&E  are  redundant  with  technical  issues  tested  in  the 
SPEARS  CBT  T&E.  For  these  issues,  the  data  resulting  fi'om  the  CBT  T&E  will  be  incorporated 
into  the  SPEARS  TIP  Test  and  Evaluation  Report  (TER),  with  the  source  of  the  data 
appropriately  annotated.  All  technical  issues  are  restated  here  for  clarity. 

The  following  details  the  CTICs  and  associated  MOPs  for  the  OT&E.  Unless  otherwise  stated, 
the  MOPs  will  be  collected  through  structured  protocols  conducted  in  either  the  Aviation  Security 
Laboratory  or  on-site  at  the  airport.  The  SPEARS  device  will  be  made  operational  at  the  testing 
site.  Any  training  required  on  the  operation  of  applicable  system  components  will  be  requested 
from  the  vendor.  The  researcher  will  exercise  the  system  to  the  extent  necessary  to  evaluate  the 
issue. 


10 


1.9.1  Image  Content. 


1.9. 1.1  Issue. 


Do  images  represent  the  range  of  current  threats? 

Criterion  A.  At  least  300  different  CTIs  will  be  available  representing  threat  categories  of 
weapons,  incendiaries,  explosives,  and  other  dangerous/deadly  devices. 

MOP  9.  The  number  of  CTIs  available. 

MOP  10.  The  threat  category  of  each  CTI  available. 

Criterion  B.  At  least  300  different  FTIs  will  be  available  representing  threat  categories  of 
weapons,  incendiaries,  explosives,  and  other  dangerous/deadly  devices. 

MOP  11.  The  number  of  FTIs  available. 

MOP  12.  The  threat  category  of  each  FTI  available. 

Criterion  C.  At  least  two  different  aspect  angles  will  be  available  for  each  particular  threat  object. 

MOP  13.  The  aspect  angles  available  for  each  CTI. 

MOP  14.  The  aspect  angles  available  for  each  FTI. 

Criterion  D.  The  degree  of  detection  difficulty  varies. 

MOP  15.  Detection  difficulty  of  the  FTIs  varies  as  a  function  of  the  contents  of  the  actual 
bag  being  examined  and  will,  therefore,  not  be  assessed.  CTIs  will  be  assessed  by  ratings 
provided  by  screeners  or  subject-matter  experts. 

1.9. 1.2  Test  Strategy. 


An  inspection  of  the  threat  object  database  will  be  made  to  verify  that  at  least  300  different  CTIs 
and  300  FTIs  are  available,  and  that  these  are  representative  of  the  threat  categories  of  weapons, 
incendiaries,  explosives,  and  other  dangerous/deadly  devices.  The  inspection  will  also  determine 
that  at  least  two  different  aspect  angles  are  available  for  each  CTI  and  FTI  and  that  the  degree  of 
detection  difficulty  varies  for  these  different  items  in  CTI  displays.  These  evaluations  will  be 
accompanied  by  completing  the  SPEARS  Image  Content  Checklist  (Appendix  D),  which 
addresses  the  pertinent  technical  issues. 
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1.9.2  Customization. 


1.9.2. 1  Issue. 


Can  performance  monitoring  be  tailored  to  individual  screener  needs? 

Criterion  A.  Images  can  be  selected  for  each  screener  as  a  function  of  threat  category,  difficulty, 
and  aspect  angle,  which  are  archived  along  with  the  associated  outcome  of  the  FTI  (hit,  miss, 
false  alarm,  correct  rejection). 

MOP  16.  The  methods  of  selecting  CTIs  based  on  image  factors. 

MOP  17.  The  methods  of  archiving  results  for  CTIs  based  on  image  fectors. 

MOP  18.  The  methods  of  selecting  FTIs  based  on  image  factors. 

MOP  19.  The  methods  of  archiving  results  for  FTIs  based  on  image  factors. 

Criterion  B.  FTIs  and  CTIs  can  be  automatically  or  manually  selected  for  presentation  based  on 
such  factors  as  time  of  day,  checkpoint  activity,  or  screener  identity. 

MOP  20.  The  methods  of  selecting  CTIs  based  on  external  factors. 

MOP  21.  The  methods  of  selecting  FTIs  based  on  external  factors. 

1. 9.2.2  Test  Strategy. 


During  the  OT&E,  the  SPEARS  Customization  Checklist  (Appendix  E)  will  be  used  to  assess  the 
degree  to  which  TnT™  performance  monitoring  has  been  tailored  to  individual  screener  needs. 
The  checklist  items  cover  the  methods  for  selecting  and  archiving  of  CTIs  and  FTIs  using  both 
manual  and  automatic  processes. 

1.9.3  Feedback. 


1.9.3. 1  Issue. 


Is  feedback  provided? 

Criterion  A.  The  device  will  immediately  inform  the  screener  on  whether  each  threat  object  was 
correctly  identified. 
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MOP  22.  The  feedback  provided  by  the  SPEARS  device  regarding  the  accuracy  of  threat 
object  disposition  as  determined  by  demonstration  and  inspection. 

1.9.3 ,2  Test  Strategy. 


Test  bags  not  containing  actual  threat  objects  will  be  passed  through  the  X-ray  scanning  device. 
The  SPEARS  device  will  be  used  to  insert  threat  object  images  onto  the  test  bag  images.  In 
addition,  test  bags  will  be  run  through  without  receiving  FTI  projections.  The  researcher  will 
correctly  or  incorrectly  press  the  TIP  key  when  an  image  is  presented  and  record  the  feedback 
provided.  For  the  purposes  of  this  test,  feedback  is  defined  as  any  combination  of  visual  or 
auditory  messages  that  is  intended  to  alert  the  X-ray  screener  and  explain  the  status  of  the  image 
scanning.  Additionally,  the  checklist  (Appendix  F)  will  be  used  to  determine  the  extent  to  which 
screener  feedback  is  provided. 

1.9.4  Capability  Summaries. 


1. 9.4.1  Issue. 


Are  performance  reports  prepared? 

Criterion  A.  The  device  automatically  generates  performance  reports  containing  descriptive 
statistics  and  readily  understandable  interpretations  to  summarize  records  of  threats  presented  and 
the  outcomes. 

MOP  23.  The  content  and  presentation  method  of  summary  reports  provided  by  the 
SPEARS  device  as  determined  by  demonstration  and  inspection. 

MOP  24.  The  degree  to  which  reports  are  determined  to  be  understandable  through 
interviews  with  screeners  and  supervisors  and  assessment  using  Human  Factors  (HF)  checkhst 
design  principles  and  criteria. 

Criterion  B.  Supervisors  can  be  alerted  when  screener  performance  falls  below  operational 
requirements  during  a  screening  session. 

MOP  25.  The  performance  tracking  and  notification  parameters  provided  by  the  SPEARS 
device  as  determined  by  description  and  inspection. 

1.9. 4.2  Test  Strategy. 


The  performance  reports  will  be  automatically  tabulated  and  produced  by  the  SPEARS  device 
upon  prompting  by  the  researcher.  A  SPEARS  CapabUities  Summaries  Checklist  (Appendbc  G) 
will  also  be  used  to  evaluate  the  content  and  presentation  methods  employed  in  the  performance 
summary  reports. 
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1.9.5  Interoperability. 


1.9. 5.1  Issue. 


Can  the  TnT™  equipment  communicate  with  remote  computers? 

Criterion  A.  Additional  FTIs  and  CTIs  can  be  received  from  remote  sites. 
MOP  26.  The  method  of  remote  connection  to  the  SPEARS  device. 
MOP  27.  Demonstration  of  CTIs  downloaded  to  the  SPEARS  device. 
MOP  28.  Demonstration  of  FTIs  downloaded  to  the  SPEARS  device. 
MOP  29.  Time  required  to  transfer  a  CTI. 

MOP  30.  Time  required  to  transfer  an  FTI. 

Criterion  B.  The  equipment  will  be  able  to  transmit  reports  to  remote  sites. 

MOP  31.  The  SPEARS  remote  performance  monitoring  capability. 

1.9. 5. 2  Test  Strategy. 


The  Interoperability  Checklist  (Appendix  H)  will  be  used  to  evaluate  the  extent  to  which  the 
TnT™  equipment  can  communicate  with  remote  computers.  A  computer  will  be  installed  at  a 
remote  location  and  coimected  by  modem  with  the  device  at  the  test  site.  A  researcher  at  each 
location  will  attempt  to  download  new  images  to  the  SPEARS  devices.  Additionally,  researchers 
will  attempt  to  access  reports  from  the  remotely  situated  SPEARS  device. 

1.9.6  Security. 


1.9.6. 1  Issue. 


Is  access  restricted? 

Criterion  A.  Only  authorized  screeners  and  supervisors  can  access  certain  aspects  of  the  system. 
MOP  32.  Computer  Software  Security  Checklist  results. 
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1.9.6.2  Test  Strategy. 


The  security  issue  will  be  investigated  though  a  combination  of  a  structured  checklist  and  a 
demonstration  of  security  features.  The  security  checklist  is  based  on  the  Department  of  Defense 
Trusted  Computer  System  Evaluation  Criteria  (DOD-5200.28-STD)  and  is  contained  in 
Appendbc  I. 

1.9.7  Insertion. 


1. 9.7.1  Issue. 


Can  images  consisting  of  an  FTI  or  a  CTI  be  automatically  and  impredictably  (for  screeners) 
inserted  on  the  checkpoint  X-ray  display?  Each  FTI  consists  of  only  a  single  threat  object. 

Criterion  A.  FTI  position  can  be  randomly  determined. 

Criterion  B.  FTI  orientation  within  each  bag  is  controllable. 

MOP  33.  Recording  of  presentation  order  for  FTIs  and  CTIs,  and  image  threat  placement 
for  FTIs,  using  a  baggage  content  locator  matrix,  which  will  be  used  to  determine  any  predictable 
presentation  patterns  and  FTI  orientation  controllability. 

1.9  7.2  Test  Strategy. 


Baggage  not  containing  threat  objects  will  be  passed  through  the  X-ray  device  and  the  SPEARS 
device  will  be  used  to  insert  FTIs  onto  the  display  of  the  bag  image.  In  addition,  a  series  of 
neutral  bags  will  be  run  through  the  X-ray  device  and  the  SPEARS  device  will  be  used  to  insert 
CTIs  into  the  baggage  flow.  For  the  bags  that  have  FTIs  inserted,  the  position  of  each  threat 
image  will  be  noted  on  a  3  by  3  cell  grid.  The  order  of  placement  in  each  cell  across  FTIs  will  be 
analyzed  for  randomness.  In  addition,  the  placement  of  CTIs  and  FTIs  within  the  baggage  flow 
will  be  recorded  and  analyzed  for  randomness.  An  Image  Insertion  Checklist  (Appendbc  J)  will 
be  used  to  evaluate  the  proper  insertion  of  images  into  the  checkpoint  X-ray  display. 

1.10  OPERATIONAL  TEST  AND  EVALUATION  MILESTONES 


Table  1  shows  the  milestones  for  planning  and  reporting  the  OT&E. 


TABLE  1.  OPERATIONAL  TEST  AND  EVALUATION  MILESTONES 


Milestone 

Date 

Responsible  Organization 

Test  Concept/Design  Approval 

03/15/95 

AAR-510 

Initial  TEP  Submittal 

05/01/95 

Contractor 

Coordinate  Test  Site 

04/03-05/95 

AAR-5 1 0/Contractor 
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X-Ray  Machine  Modifications 

07/13/95 

Contractor 

TEP  Approved 

07/14/95 

AAR-510 

Pilot  Test 

07/17-19/95 

AAR-5 1 0/Contractor 

OT&E  Initiated 

07/31/95 

AAR-5 1 0/Contractor 

TIP  1 

08/04/95 

AAR-5 1 0/Contractor 

First  Posttest 

09/12/95 

AAR-5 1 0/Contractor 

TIP  2 

09/18/95 

AAR-5 1 0/Contractor 

Second  Posttest 

11/06/95 

AAR-5 1 0/Contractor 

OT&E  Completed 

11/10/95 

AAR-5 1 0/Contractor 

TER  Submitted 

01/12/96 

Contractor 

Evaluation  Briefing 

01/26/96 

AAR-510 

2.  OPERATIONAL  TEST  AND  EVALUATION. 
2.1  SUBJECTS. 


Two  groups  of  subjects,  a  training  group  and  a  control  group,  will  be  used  to  conduct  the 
operational  assessment  of  the  SPEARS  on-line  training  and  testing.  The  control  group  will  not 
receive  SPEARS  TIP  training  during  the  OT&E.  If  the  control  group  is  aware  of  the  activities  of 
the  training  group,  the  members  might  exhibit  decreased  performance  over  time  caused  by  not 
being  selected  for  special  treatment.  To  reduce  this  effect,  the  training  and  control  groups  wiU  be 
physically  separated  by  conducting  testing  at  LAX  in  two  separate  terminals. 

The  initial  sample  will  consist  of  30  baggage  screeners  selected  from  the  day  (0630-1500  hours) 
and  swing  (1430-2300  hours)  shifts  at  LAX  Terminal  7  (United  Airlines)  and  24  screeners 
selected  from  the  day  and  swing  shifts  at  LAX  Terminal  2  (Northwest  Airlines).  The  number  of 
subjects  for  each  group  was  chosen  to  account  for  anticipated  attrition.  It  is  expected  that  20 
screeners  will  remain  in  each  group  at  the  end  of  the  study.  The  Terminal  7  screeners  will 
participate  in  the  control  group  and  the  Terminal  2  screeners  will  participate  in  the  training  group. 
To  help  control  X-ray  screening  aptitude  across  treatment  groups,  the  training  and  control  group 
samples  will  be  tested  on  the  Embry-Riddle  Aeronautical  University  (ERAU)  computerized 
versions  of  the  Hidden  Figures  Test  (HFT)  and  Hidden  Patterns  Test  (HPT)  (Gibb,  Banaijee, 
Bennett,  Baker,  and  Kelly,  1995,  and  Appendbc  K).  The  tests  will  be  used  to  form  three  blocl^g 
levels  of  high,  medium,  and  low  performance,  with  roughly  equal  numbers  of  subjects  assigned  to 
each  blocking  level  within  each  experimental  group 

2.2  SUBJECT  TRAINING. 


The  screeners  will  be  briefed  about  the  study  in  terms  of  their  activities,  schedules,  and 
expectations  as  participants.  They  will  also  be  asked  to  complete  a  consent  form  and  a  personal 
data  form  (see  Appendixes  L  and  M)  before  participating  in  the  study.  The  screeners  will  also 
receive  protocol  instructions  (see  Appendix  N)  before  the  study  and  immediately  before  each 
major  test  activity. 


16 


Because  training  and  experience  will  vary  across  subjects,  all  participants  will  attend  a  lecture 
session  that  will  provide  explanations  of  X-ray  screening  philosophy  and  procedures,  image 
interpretation,  and  possible  threat  objects.  This  training  will  seek  to  equalize  knowledge  across 
subjects.  The  training  is  expected  to  require  approximately  2  hours  to  complete. 

Trmning  on  the  use  of  the  specific  SPEARS  device  will  be  provided  to  the  entire  Terminal  2 
screening  staff  Because  of  the  flexibility  inherent  in  airport  baggage  screener  duty  cycles,  there  is 
a  chance  that  a  Terminal  2  screener  may  inadvertently  receive  a  TIP,  even  though  the  screener  is 
not  a  member  of  the  training  group.  To  account  for  this  eventuality,  all  Terminal  2  screeners  will 
receive  training  to  recognize  and  properly  resolve  a  TIP. 

2.3  TEST  ORGANIZATION. 


2,3.1  Personnel  Requirements. 


The  number  of  test  administrators  required  varies  across  testing  activities,  as  shown  in  Tables  2 
through  4.  A  detailed  description  of  the  duties  and  schedules  for  each  test  administrator  is 
included  in  Appendk  O. 


TABLE  2.  PRETEST  ADMINISTRATORS 


Personnel  No. 

Title 

Agency 

1 

Test  Director  a.m. 

FAA 

2 

Test  Director  p.m. 

FAA 

3 

Test  Manager 

GSC 

4 

Trainer  a.m. 

GSC 

5 

Trainer  p.m. 

GSC 

1 

Terminal  7  Data  Collector  a.m. 

FAA 

2 

Terminal  7  Data  Collector  p.m. 

FAA 

4 

Terminal  2  Data  Collector  a.m. 

GSC 

5 

Terminal  2  Data  Collector  p.m. 

GSC 

TABLE  3.  FIRST  POSTTEST  ADMINISTRATORS 


Personnel  No. 

Title 

Agency 

1 

Test  Director  a.m. 

FAA 

2 

Test  Director  p.m. 

FAA 

3 

Test  Manager 

GSC 

1 

Terminal  7  Data  Collector  a.m. 

FAA 

2 

Terminal  7  Data  Collector  p.m. 

FAA 

4 

Terminal  2  Data  Collector  a  m. 

GSC 

5 

Terminal  2  Data  Collector  p.m. 

GSC 
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TABLE  4.  SECOND  POSTTEST  ADMINISTRATORS 


Personnel  No. 

Title 

Agency 

1 

Test  Director  a.m. 

FAA 

2 

Test  Director  p.m. 

FAA 

3 

Test  Manager 

GSC 

1 

Data  Collector  lED  a.m. 

FAA 

2 

Data  Collector  lED  p.m. 

FAA 

4 

Data  Collector  TnT™  a.m. 

GSC 

5 

Data  Collector  TnT™  p.m. 

GSC 

2.4  TEST  ORGANIZATION  TRAINING. 


The  test  organization  will  be  trained  on  the  candidate  SPEARS  device  before  the  OT&E.  The 
test  organization  will  also  receive  a  one-day  training  session  on  test  procedures  and  protocol 
before  the  OT&E.  The  training  will  cover  data  collection  procedures  and  methodology  and  the 
procedures  to  be  followed  in  case  of  an  emergency. 

2.4.1  Operational  Test  Procedures. 


2.4. 1.1  Overview. 


The  operational  test  comprises  five  major  activities,  including  a  pretest  baseline  performance 
measurement,  an  initial  4-week  period  during  which  the  training  group  screeners  will  receive 
SPEARS  TIP,  a  first  posttest  performance  measurement,  a  second  8-week  SPEARS  TIP  period, 
and  a  second  posttest  performance  measurement.  The  schedule  for  these  activities  is  included  in 
Appendkes  P,  Q,  and  R. 

Five  site  visits  to  LAX  will  be  required  to  complete  the  operational  testing.  An  initial  orientation 
visit  will  be  conducted  to  provide  briefings  and  coordinate  test  activity  logistics  with  airline, 
airport,  and  screening  company  representatives.  A  pilot  study  will  be  conducted  prior  to  the  start 
of  test  activities  to  ensure  all  agencies  involved  in  the  test  are  adequately  prepared.  Finally,  three 
visits  will  be  required  to  conduct  the  pretest  and  two  posttests. 

2.4. 1 .2  Test  Readiness  Review. 


A  Test  Readiness  Review  (TRR)  will  be  held  prior  to  the  start  of  the  OT&E  (see  SPEARS  FIP 
OT&E  Master  Schedule,  Appendix  P).  The  review  will  assess  the  overall  status  of  the  system  and 
resolve  any  open  issues  before  beginning  the  test.  The  TRR  will  include  an  assessment  of  the  test 
cases,  test  procedures,  test  site  environment,  test  equipment,  support  software,  test  participant’s 
roles  and  responsibilities,  and  data  collection  methodology  and  tools.  Completion  of  the  TRR  will 
result  in  a  go/no-go  decision  by  the  FAA  Program  Manager  for  the  start  of  the  OT&E. 
Additional  TRRs  may  be  scheduled  as  necessary. 
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2.4. 1.3  Pilot  Study. 


A  pilot  study  will  be  conducted  approximately  2  weeks  before  the  OT&E  to  verify  the  test 
protocol  and  data  collection  procedures.  Data  collection,  the  SPEARS  TIP,  and  lED  detection 
testing  procedures  will  be  verified.  Two  screeners  will  be  required  for  the  pilot  study,  one  fi'om 
each  of  the  treatment  groups. 

2.4. 1 .4  Operational  Test  Protocol. 


The  protocol  describes  the  manner  in  which  data  will  be  gathered  to  assess  the  operational 
effectiveness  of  SPEARS  TIP.  The  specific  protocol  for  each  of  these  data  collection  exercises  is 
described  in  sections  2.4.2  through  2.4.2. 1.2. 

Screening  activities  at  LAX  are  carried  out  24  hours  per  day,  and  are  divided  into  three  shifts.  To 
obtain  the  required  screener  sample  size,  OT&E  activities  will  be  carried  out  during  the  morning 
shift  (0630-1500  hours)  and  the  swing  shift(1430-2300).  The  two  shifts  overlap  for  30  minutes 
between  1430-1500  hours  to  ensure  a  smooth  transition  between  shifts.  During  the  transition,  the 
swing  shift  operates  the  screening  stations  and  is  monitored  by  the  morning  shift.  Because  data 
must  be  collected  during  two  shifts,  there  are  two  equivalent  data  collection  cycles  during  each 
day. 

All  briefing  and  data  collection  activities  will  take  place  in  Room  2034  and  at  the  Security 
Checkpoint  of  Terminal  2  and  in  the  Aloha  Room  of  Terminal  7.  Diagrams  of  the  terminals  and 
the  rooms  are  provided  in  Appendix  S. 

2.4.2  Pretest. 


The  pretest  will  be  conducted  to  assess  the  baseline  lED  detection  performance  of  the  screeners 
participating  in  the  study.  Schedules  for  the  pretest  are  included  in  Appendixes  Q  and  R.  Both 
the  training  and  control  groups  will  be  pretested  concurrently.  The  geographic  separation 
between  the  training  and  control  groups  will  require  two  sets  of  test  administrators,  one  per 
group. 

Data  will  be  collected  on  days  one  through  three  of  the  pretest.  Each  screener  in  each  treatment 
group  will  be  required  once  for  a  2-hour  period.  Two  screeners  will  be  tested  simultaneously  in 
each  treatment  group.  Participants  will  be  administered  the  Snellen  equivalent  High  (96  percent) 
Contrast  Vision  Acuity  Test  (see  Appendix  T  for  the  score  sheet)  and  the  ERAU  HFT  and  HPT. 
These  three  activities  will  require  approximately  60  minutes  to  complete. 

Because  the  training  and  control  groups  will  be  matched  on  the  HFT  and  HPT  scores,  it  may  be 
necessary  to  pretest  more  than  the  number  of  screeners  required  for  each  treatment  group.  Day  4 
of  the  pretest  is  provided  for  this  contingency.  Following  the  HFT  and  HPT  and  selection  as  a 
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study  participant,  each  screener  will  be  assigned  a  subject  number  which  will  be  used  to 
anonymously  identify  the  screener  during  the  study. 

Participants  will  then  receive  the  computerized  lED  detection  performance  test.  Before  beginning 
the  performance  measurement,  the  data  collector  will  brief  each  screener  using  the  instructions 
contained  in  Appendixes  N,  U,  V,  and  W.  Each  screener  will  be  informed  that  simulated 
explosive  devices  will  be  present  in  some  of  the  bag  images  to  be  screened  during  the  test  trial. 
The  screeners  will  be  unaware  of  the  physical  appearance  of  the  original  bags  or  their  insertion 
order.  Screeners  will  carry  out  normal  screening  operations  and  will  not  receive  operational 
direction  from  the  test  administrators. 

This  test  will  require  approximately  35  minutes.  Before  each  test  session,  the  FAA  data  collector 
will  assign  an  operator  identification  number  to  each  screener  for  access  to  the  test  device  and  will 
ensure  that  the  test  device  is  operable  and  properly  configured.  If  the  test  device  malfunctions 
during  the  test  trial,  the  data  collector  will  inform  the  test  manager,  who  will  arrange  for 
maintenance.  If  the  malfunction  cannot  be  resolved  within  a  relatively  short  time,  the  test  will  be 
postponed  until  the  test  device  is  operable. 

The  data  collector  will  start  the  test  trial  and  ask  the  screener  to  begin  the  test.  The  testing  device 
will  automatically  display  the  first  bag  image  to  the  screener.  The  screener’ s  task  is  to  indicate 
whether  the  bag  contains  a  threat  image  by  pressing  the  appropriate  key  on  the  keyboard.  After 
the  screeners  have  responded  to  this  initial  question,  they  will  be  asked  a  second  question  based 
upon  their  response  to  the  first  question.  They  will  be  asked  to  indicate  how  confident  they  were 
in  their  original  response  concerning  the  presence  or  absence  of  an  LED  in  the  bag.  The  testing 
device  will  record  the  response  and  automatically  forward  to  the  next  image.  The  data  collector 
will  stop  the  test  when  all  bag  images  have  been  presented. 

Each  computerized  test  will  consist  of  25  test  bag  images  containing  MBSs  within  a  series  of  200 
CO  comparison  bag  images.  A  description  of  the  test  bag  configuration  is  included  in  Appendkes 
X  and  Y.  All  screeners  in  the  study  will  view  the  same  test  and  comparison  bag  images.  To 
control  for  presentation  order,  sk  randomly  generated  presentation  orders  for  the  200  bag  images 
will  be  used.  The  screeners  in  the  training  group  will  be  arranged  into  groups  of  four.  The 
screeners  in  the  control  group  will  be  arranged  into  groups  of  five.  All  groups  in  each  of  the 
training  and  control  groups  will  receive  one  of  these  sk  presentation  protocols. 

The  control  group  will  be  returned  to  normal  duties  after  the  lED  test.  Instruction  in  the 
operation  of  the  SPEARS  TIP  training  equipment  and  the  procedure  for  threat  resolution  when 
the  TnT  TIP  feature  is  operational  will  be  provided  to  the  entire  Terminal  2  staff.  The  training 
group  is  a  subset  of  the  total  screening  staff.  However,  all  receive  TIP  equipment  training 
because  FTI  presentation  can  only  be  controlled  in  terms  of  time  and  not  on  an  individual  screener 
basis.  It  is  therefore  possible  that  a  screener  outside  of  the  training  group  will  accidentally  receive 
a  TIP.  Screeners  in  the  training  group  will  then  again  be  briefed  on  the  training  that  v^l  occur 
during  the  balance  of  the  OT&E.  The  participants  will  then  be  returned  to  their  normal  duties. 
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2.4.2. 1  On-line  Testing. 


The  basic  task  of  the  X-ray  screener  is  to  monitor  X-ray  images  presented  on  display  screens  and 
identify  potential  threat  objects  within  the  images.  The  task  for  both  SPEARS  TIP  and  lED 
detection  testing  will  match  this  basic  operational  task;  Screeners  will  examine  bags  and  identify 
those  that  contain  a  potential  threat  object. 

The  SPEARS  TIP  task  involves  the  X-ray  screeners  carrying  out  normal  screening  duties  at  their 
usual  screening  station  (Terminal  2  checkpoint).  TnT™  devices  will  be  connected  to  two  of  the 
three  X-ray  machines  at  the  Terminal  2  checkpoint,  as  depicted  in  Appendix  S.  The  TnT™ 
devices  will  be  configured  to  have  the  TIP  function  insert  up  to  24  FTIs  into  the  normal  passenger 
baggage  flow  to  be  observed  by  each  screener  on  a  given  day. 

The  method  for  FTI  selection  and  insertion  is  as  follows.  The  number  of  FTIs  to  be  presented  to 
each  screener  on  a  given  day  is  detailed  in  the  randomly  generated  protocol  shown  in  Appendbc  Z. 
During  each  8-hour  shift,  the  duty  cycle  for  a  screener  normally  consists  of  a  20-minute  period  on 
the  X-ray  equipment,  20  minutes  waning,  20  minutes  performing  bag  searches,  and  20  minutes 
ofiF.  A  normal  shift  also  includes  a  30-minute  lunch  break.  This  duty  cycle  results  in  each  screener 
being  on  the  X-ray  equipment  a  maximum  of  once  per  hour,  and  a  total  of  approximately  four 
sessions  per  shift.  Because  of  the  inherent  flexibility  of  screening  operations,  it  is  impossible  to 
determine  when  these  four  sessions  will  occur  for  each  screener.  For  this  reason,  each  hour  of 
each  shift  will  be  divided  into  three,  20-minute  segments.  The  number  of  FTIs  to  be  presented  to 
a  screener  on  a  given  day  will  be  doubled  and  spread  across  the  eight-hour  shift.  The  number  of 
FTIs  to  be  presented  in  each  hour  will  be  replicated  in  each  20-minute  segment.  Using  this 
procedure,  reasonable  assurance  is  provided  that  each  screener  will  receive  close  to  the  required 
number  of  FTIs  and  that  all  screeners  will  receive  approximately  the  same  FTI  exposure  on  a 
given  day.  The  flight  schedule  for  Terminal  2  is  included  in  Appendbc  AA.  The  ebb  and  flow  of 
passengers  will  result  in  uneven  FTI  presentation  rates  across  screeners.  This  is  unavoidable 
given  the  operational  environment  and  the  way  the  device  can  be  programmed. 

When  the  X-ray  screener  believes  a  threat  object  is  present  in  a  bag  (in  this  case,  an  FTI),  the 
screener  will  press  the  TnT  TIP  button  to  probe  the  SPEARS  device  for  feedback.  If  the  bag 
contains  an  FTI,  the  SPEARS  device  will  confirm  the  threat  object  and  indicate  it  on  the  display. 
The  threat  object  vwU  then  be  erased  from  the  screen,  allowing  the  X-ray  screener  to  ensure  that 
there  are  no  additional  real  threat  objects  in  the  bag.  Only  one  FTI  per  bag  will  be  generated  by 
the  SPEARS  device,  but  the  possibility  exists  that  there  is  an  actual  object  in  the  bag.  If  the  bag 
does  not  contain  an  FTI,  but  the  screener  indicates  a  possible  threat  through  the  TIP  key,  then  the 
bag  is  recorded  as  a  false  alarm,  notification  is  given  to  the  screener  and  supervisor,  and  normal 
security  procedures  are  followed  (see  Appendbc  W.  If  the  screener  fails  to  detect  the  FTI,  the 
TnT  will  notify  the  screener  of  the  miss  and  record  the  presentation  in  the  screener’ s  database. 


2.4.2. 1 . 1  First  Posttest. 
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The  first  posttest  will  consist  of  lED  detection  testing  only.  This  testing  will  follow  the  same 
protocol  contained  in  the  pretest  description.  Each  screener  will,  therefore,  be  required  to  take  a 
test  session  lasting  approximately  35  minutes.  The  first  posttest  will  require  2  days  to  complete 
(see  Appendix  Q).  The  control  group  and  training  group  will  be  tested  concurrently 


2.4.2. 1 .2  Second  Posttest. 

The  second  posttest  will  again  involve  a  35-minute  lED  detection  test  session  for  each  screener. 
Following  FED  detection  testing,  each  screener  in  the  training  group  will  complete  a  usability 
assessment  form.  The  second  posttest  will  require  3  days  to  complete  (see  Appendbc  Q).  The 
training  group  and  control  group  will  be  tested  concurrently. 

2.4.3  SPEARS  Usability  Assessment  Methodology. 


The  tailored  checklist  will  be  completed  during  the  testing  of  the  technical  issues. 

The  subjective  ratings  will  concern  the  SPEARS  operating  procedures  and  feedback  provided  by 
the  SPEARS  device  during  real  time  baggage  screening.  The  ratings  will  be  provided  through  5- 
point  Likert  scales  (see  Appendbc  BB)  developed  for  the  SPEARS  training  and  testing  and 
administered  to  each  screener  in  the  training  group  during  the  second  posttest  session. 

The  structured  interview  (see  Appendbc  C)  will  be  conducted  to  probe  the  usability  of  the 
SPEARS  device  and  'will  be  complementary  to  the  Likert  scale  ratings. 

Any  procedural  errors  made  by  the  training  group  when  operating  the  SPEARS  equipment  will  be 
noted. 

2.4.4  SPEARS  Insertion  Representativeness  Methodology. 


The  insertion  representativeness  analysis  will  be  conducted  by  having  each  control  group  screener 
view  images  of  baggage  with  inserted  FTIs,  images  of  CTIs,  images  of  baggage  containing  actual 
threats,  and  images  of  baggage  containing  no  threat  objects.  Each  image  will  be  rated  as  to  its 
fidehty;  for  example,  the  degree  to  which  the  image  appears  like  an  image  that  screeners  might 
encounter  in  performing  their  normal  job  functions.  In  addition,  the  physical  properties  of  each 
image  will  be  rated,  including  the  density  of  the  objects  packed  in  the  bags,  the  type  of  items 
packed  in  the  bags,  the  order  of  placement  of  items  in  the  bags,  and  the  type  of  bag  being 
presented.  Each  of  these  items  will  be  rated  on  a  seven-point  Likert  scale  ranging  from  “Would 
Never  Encounter”  to  “Very  Likely  to  Encounter”  a  bag  in  the  real  world  (see  Appendbc  CC). 
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2.5  DATA. 


2.5.1  SPEARS  TIP  Training  and  Testing  Data. 


For  each  of  the  pretest  and  the  two  posttests,  the  screener  decision  and  confidence  responses 
observed  while  scanning  for  simulated  lEDs  in  the  computerized  passenger  bag  X-ray  images  wall 
be  collected  for  each  screener  in  both  treatment  groups.  During  the  TIP  training  periods,  data 
wall  be  collected  on  the  number  of  TIP  hits  and  misses  for  each  screener. 

2.5.2  SPEARS  Usability  Data. 


The  data  resulting  fi-om  the  SPEARS  usability  analysis  will  be  the  HFE  checklist  data,  the  Likert 
scale  responses,  the  responses  to  the  structured  interview,  and  any  procedural  errors  noted  when 
operating  the  SPEARS  device  for  each  subject  in  the  training  group. 

2  5  3  Insertion  Representativeness  Data. 


The  data  resulting  fi’om  the  insertion  representativeness  assessment  will  be  the  Likert  scale 
responses  for  each  screener  in  the  control  group. 

2.6  TEST  DATA  MANAGEMENT. 


The  final  operational  test  database  (shown  in  Table  5)  will  contain  the  data  required  to  statistically 
analyze  the  critical  operational  issues. 

All  technical  and  operational  data  will  be  retained  by  the  principal  investigator  for  the  project  for  5 
years.  In  accordance  with  professional  and  ethical  standards,  the  principal  investigator  will 
maintain  separate  records  of  performance  data  and  the  names  of  participating  screening  personnel. 
A  complete  set  of  data  will  be  provided  to  the  FAA  Aviation  Security  Laboratory  for  storage  as 
well,  without  the  associated  personnel  records. 
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TABLE  5.  SCREENER  PROFICIENCY  EVALUATION  AND  REPORTING  SYSTEM  ON¬ 
LINE  OPERATIONAL  TEST  DATABASE 


Screener  ID 

Screener  Characteristics  Check 
Experience 

X-ray  Equipment  Experience 
Type  of  X-ray  Equipment  Used 
lED  Test  Date 
Vision  Test  Score 
ERAU  HFT  Scores 
ERAU  HPT  Scores 
SPEARS  FIP  Performance 
(lED  Decision/Confidence  Rating)  Scores 
Bag  Number  for  Each  Response  and  Confidence  Rating 
Usability  Likert  Scale  Ratings 
Usability  Structured  Interview  Results 
SPEARS  Procedural  Errors 
Image  Representativeness  Likert  Scale  Ratings 


2.1  DATA  ANALYSIS. 


2.7.1  Experimental  Design. 


The  experimental  design  for  the  operational  test  will  be  an  untreated  control-group  design  with 
pretest  and  posttest  measures  (Cook  and  Campbell,  1979),  which  will  allow  for  a  direct 
assessment  of  the  effects  of  the  SPEARS  TIP  intervention  on  screener  lED  detection 
performance.  The  basic  design  will  be  modified  to  include  a  second  posttest  data  collection  to 
measure  the  retention  of  the  skill  over  time.  Using  Cook  and  Campbell’s  (1979)  notation,  the 
experimental  design  is  depicted  in  Figure  1. 


tinted 


Training  Group 
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FIGURE  1.  EXPERIMENTAL  DESIGN 

Where  X  indicates  the  SPEARS  treatments  (Xi  indicates  the  EG&G  TnT™  system),  and  O, 
which  occurs  during  X,  indicates  the  ordinal  number  of  screener  performance  tests. 
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2.7. 1  ■  1  Independent  Variables. 


There  will  be  two  independent  variables  for  the  SPEARS  TIP  evaluation  in  terms  of  lED 
detection  performance.  The  independent  variable,  SYSTEM,  will  represent  the  SPEARS  device 
and  will  have  two  levels:  training  and  control.  The  other  independent  variable,  TRIAL  will 
represent  the  SPEARS  training  intervention  and  will  have  three  levels:  pretest,  first  posttest,  and 
second  posttest. 

For  the  FTI  performance  evaluation,  time  seqments  is  the  only  independent  variable.  The  data 
will  be  divided  into  weekly  segments  for  analysis. 

2.7. 1.2  Dependent  Variables. 


The  dependent  variables  will  be  derived  fi-om  the  off-line  screener  decision  of  lED  presence  for 
each  bag  and  are  the  SDT  variables  of  Pa,  Pf»,  d',  and  c.  These  variables  are  not  available  for  the 
TIP  potion  which  is  restricted  to  only  the  Pa  for  each  weekly  segment.  The  calculated  values  of 
Pa,  P&,  d',  and  c  will  be  compared  across  training  conditions. 

For  each  screener,  the  bag  decisions  will  be  corrected  to  hits  and  false  alarms  based  on  the 
correctness  of  screener  yes-no  responses  to  whether  an  lED  was  present.  Nh  and  Nf*  are  the  total 
number  of  hits  and  false  alarms  across  all  bags. 

For  each  screener,  Ni,  will  be  used  to  calculate  Pj  as  follows: 

Pd  =  Nh/NT 

where  Nj  is  the  total  number  of  test  bag  images  presented. 

Nfa  will  be  used  to  calculate  P^  as  follows: 


Pfi  =  N6/Nc 


where  Nc  is  the  total  number  of  comparison  bags  presented. 

The  confidence  scores  fi’om  the  lED  tests  will  be  used  to  plot  a  Receiver  Operating  Characteristic 
(ROC)  curve  fi’om  which  a  value  of  d'  will  be  determined.  A  rating  scale  technique  (Green  and 
Swets,  1966)  will  be  used,  and  will  employ  four  confidence  rating  categories  as  follows: 

Category  1  -  yes  -  more  sure 
Category  2  -  yes  -  less  sure 
Category  3  -  no  -  more  sure 
Category  4  -  no  -  less  sure 

These  will  provide  the  points  needed  to  plot  the  ROC  curve. 
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A  value  of  c  is  the  distance  from  the  intersection  of  the  signal-plus-noise  (SN)  and  noise  (N) 
distributions  measured  in  z-score  unites.  The  measure  c  is  computed  as  follows: 

C  “  .5(Zfa  Zh) 

After  completing  the  lED  test  trials,  the  first  step  in  the  analysis  will  be  to  separate  the  MBS  bag 
responses  from  the  comparison  bag  responses  and  to  total  these  separately  for  each  subject 
according  to  the  rating  response  each  trial  received  (see  Figure  2).  The  next  step  will  be  to 
convert  the  raw  data  into  a  set  of  hit  and  false  alarm  rates.  This  will  involve  starting  with  the 
strictest  response  category  (1)  and  determining  a  Pa  and  Pf*  value  from  the  cell  data.  Two 
additional  Pa  and  Pf,  values  are  determined  in  a  similar  manner  by  collapsing  the  data  across 
categories  1  and  2  and  categories  1,  2,  and  3.  To  obtain  the  points  for  the  ROC  curve,  the  values 
of  Pa  and  Pa  will  then  be  converted  into  Zh  and  Z^  by  using  the  normal  tables  and  plotting  against 
one  another.  A  linear  regression  will  then  be  determined  through  the  three  points.  A  value  of  d' 
will  be  determined  from  the  point  where  the  negative  diagonal  intersects  this  regression  line. 
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FIGURE  2.  RATING  SCALE  CALCULATIONS  OF  Pa  AND  P^ 

2.7. 1 .3  Controlled  Variables. 


TIP  EXPOSURE:  This  variable  is  the  rate  that  FTI  bags  will  be  presented  in  the  normal 
passenger  baggage  flow.  The  Tff  EXPOSURE  will  be  used  during  SPEARS  training  and  set  to  a 
maximum  of  24  TIPs  and  a  minimum  of  2  TIPs  for  each  screener  shift.  The  sequence  of  TIPs 
within  the  normal  passenger  bag  flow  will  be  randomized.  Screeners  will  receive  feedback  on 
detection  performance. 

2.7. 1 .4  Predicted  Results. 


2.7. 1 .4. 1  Training  Effects. 

Two  hypotheses  can  be  tested  to  determine  whether  the  SPEARS  is  operationally  effective  for  the 
development  and  maintenance  of  baggage  screening  skills. 
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Hypothesis  1  Training:  Screener  performance  in  detecting  lEDs  will  be  significantly  improved  as 
a  result  of  SPEARS  TIP. 

Hypothesis  2:  Screener  lED  detection  proficiency  will  be  maintained  over  time  through  the  use  of 
SPEARS  TIP. 

The  first  hypothesis  requires  an  improvement  in  Pa  without  a  corresponding  increase  in  Pfa  for  the 
training  group  relative  to  the  control  group.  The  second  hypothesis  requires  that  this  performance 
effect  be  sustained  over  time.  If  this  improvement  in  performance  does  not  occur  or  is  not 
sustained  over  time,  then  the  SPEARS  is  not  operationally  effective  for  either  or  both.  Some 
research  indicates  that  the  performance  should  continue  to  gradually  increase  to  some  asymptotic 
level.  For  this  study,  however,  the  degree  of  this  gradual  increase  is  not  predictable.  A 
hypothetical  representation  of  the  hypothesized  results  is  provided  in  Figure  3. 

2.7. 1 .4.2  Vigilance  Effects. 

Similar  hypotheses  can  be  posted  for  the  effects  of  SPEARS  TIP  on  screener  vigilance,  but  they 
can  only  be  tested  over  time  with  the  training  group.  Therefore,  TIP  performance  will  be  divided 
into  weekly  segments  and  tested  for  the  following  hypotheses. 

Hypotheses  1:  Screener  performance  (Pa)  will  improve  at  detecting  TIPs  during  the  initial  weeks 
of  presentation. 

Hypothesis  2:  Once  Pa  performance  increases  to  asymlotic  levels,  it  will  not  decrease  during  the 
later  segments  of  the  study  (see  also  the  trained  performance  line  in  Figure  3). 


FIGURE  3.  HYPOTHETICAL  OPERATIONAL  TESTING  RESULTS 
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2  ■  7 . 1  ■  5  Statistical  Analysis. 


As  noted  above,  an  eflFort  will  be  made  to  avoid  contamination  of  the  control  group  by  selecting 
the  training  and  control  groups  from  two  separate  terminals  at  LAX.  This  control  strategy  does, 
however,  impose  some  penalty  on  the  statistical  analysis  of  the  resultant  data.  The  primary 
penalty  centers  on  the  assumption  of  equivalent  groups  across  the  two  treatment  groups  (Training 
and  Control)  within  the  SYSTEM  variable.  Statistical  analysis  procedures  assume  equivalent 
groups  or  a  random  assignment  of  individuals  to  groups.  Without  one  of  these  control 
procedures,  any  effects  discovered  later  in  the  study  could  be  attributed  to  the  mere  differences  in 
the  group  and  not  to  the  treatment. 

Two  approaches  that  control  for  these  biases  include  experimental  control  and  statistical  control 
(Cook  and  Campbell,  1979;  Kirk,  1982).  A  primary  technique  for  experimental  control  involves 
blocking  the  members  of  each  group  by  skill  level  using  a  predetermined  criteria.  Thus,  each 
member  of  the  training  group  would  have  an  equivalent  performing  member  in  the  control  group, 
as  determined  by  a  pretest  with  a  high  correlation  to  screening  skill,  in  this  case,  the  HFT  and 
HPT  tasks. 

This  experimental  control  method  will  be  employed  and  combined  with  a  standard  statistical 
analysis  technique.  A  3-Way  Analysis  of  Variance  (ANOVA)  design  will  be  implemented  (Kirk, 
1982),  with  the  HFT/HPT  measures  used  as  a  blocking  variable  in  the  analysis  and  the 
performance  test  measures  used  as  the  dependent  measures.  The  resulting  experimental  design 
will  allow  for  analysis  of  the  main  effect  for  SYSTEM,  the  main  effect  for  Trial,  and  the 
interaction  between  System  and  Trial.  Significant  effects  within  System,  Trial,  and  the 
System  x  Trial  interaction  will  be  investigated  using  post  hoc  Neuman-Keuls  tests.  Table  6  is 
the  ANOVA  summary  table  representing  this  analysis. 

TABLE  6.  SCREENER  PROFICffiNCY  EVALUATION  AND  REPORTING  SYSTEM 
ON-LINE  TRAINING  AND  TESTING  OPERATIONAL  TEST 
ANALYSIS  OF  VARIANCE  SUMMARY 


Source 

df 

SS 

MS 

F 

p 

Sys 

1 

SSsvs 

MSsvs 

MSsvb/MSsvsH 

H 

2 

SSh 

MSh 

MSh/MSs/svsh 

- 

SysH 

2 

SSsvsH 

MSsvbH 

MSsvsH^Ss/SiibH 

- 

S/SysH 

34 

SSs/svbh 

MSs/swh 

- 

T 

2 

SSt 

MSt 

MSt/  MStxs/svsh 

- 

T  X  Sys 

2 

SStxSvs 

MStxSvs 

MSjxSvs/MSxxSvsH 

TxH 

4 

SStxh 

MStxh 

MStxh/MStxS/SvsH 

- 

T  X  SysH 

4 

SStxSvsH 

MStxSvsH 

MStxSvsh/MStxS/SvsH 

- 

T  X  S/SysH 

68 

SStxS/SvsH 

MStxS/SvsH 

- 

Total 

119 

SSxotal 

- 

- 
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NOTE:  Sys  =  System  (TnT™,  Control)  -  Between  Groups,  fixed 

H  =  HFT/HPT  (low,  medium,  high),  Between  ftoups,  random 
T  =  Trial  (Tl,  T2,  T3),  Within  Groups,  fixed 
S  =  Subjects 

Screener  attrition  may  result  in  unequal  numbers  in  the  training  and  control  groups  at  the  end  of 
the  study.  In  this  eventuality,  an  unequal-n  ANOVA  will  be  conducted  if  these  numbers  are  not 
overly  disparate.  If  a  large  difference  exists  between  the  final  numbers  of  subjects  in  each  group, 
screeners  will  be  matched  between  groups  based  on  their  pretest  scores.  The  unmatched 
screeners  in  the  group  containing  more  subjects  will  be  excluded  fi'om  further  analysis. 

There  may  be  some  confounding  effects  due  to  repeated  exposures  of  the  participants  to  the  data 
collection  procedures  used  to  determine  performance.  For  example,  screener  performance  may  be 
improved  over  time  simply  because  the  screeners  are  more  comfortable  with  the  data  collection 
trials,  and  not  due  to  any  training  effect. 

These  controls  will  be  of  less  concern  in  analyzing  the  FTI  performance,  because  all  the  data  are 
collected  on  the  same  group.  A  one-way,  repeated  measures  ANOVA  will  be  used  to  analyze  the 
effects  of  exposure  to  FTIs  on  FTI  detection. 

2.7.2  Usability  Assessment. 


Descriptive  statistics  will  be  produced  for  the  Likert  Scale  data.  Any  Likert  rating  having  an 
average  score  of  unacceptable  will  be  deemed  to  be  a  deficiency.  Recommendations  will  be  made 
to  rectify  the  deficiencies.  Any  software,  hardware,  or  procedural  aspects  that  were  marked  as 
failures  on  the  HFE  checklist,  received  negative  comment  by  the  screeners,  or  were  noted  during 
operation  of  the  SPEARS  equipment  will  be  reported  and  recommendations  for  improvement  will 
be  made. 

2.7.3  Insertion  Representativeness  Assessment. 


For  each  assessment  category  rated,  an  average  rating  and  standard  deviation  will  be  determined 
across  screeners  for  each  image  presented.  Any  average  rating  of  3  or  less  (below  the  “Likely  to 
Occur”  behavioral  marker)  will  be  considered  to  be  a  failure  of  the  criteria  for  that  specific  rating 
category  for  that  bag  image.  Data  will  then  be  collapsed  across  bags  and  subjects  within  each 
baggage  category.  Average  ratings  and  standard  deviations  will  then  be  determined  for  each 
rating  category  within  each  bag  category.  FTI  bags  and  CTI  bags  will  be  compared  to  images 
containing  actual  threats  using  a  t-test.  Any  significant  difference  in  ratings  between  baggage 
categories  will  mean  that  the  criteria  were  not  passed  for  the  FTI  or  CTI  rating  category  in 
question. 
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3.  PRESENTATION  OF  OPERATIONAL  TEST  AND  EVALUATION  RESULTS. 


3.1  SPEARSTRAINING. 


The  results  from  the  ANOVA  carried  out  on  lED  Pd,  d',  and  c  across  time  and  training 
conditions  will  be  reported,  along  with  associated  mean  and  standard  deviation  for  each  cell.  The 
results  of  the  post  hoc  comparisons  conducted  on  any  significant  effects  will  also  be  reported. 
COIC  A  will  be  met  by  SPEARS  if  the  Pd  exhibited  for  the  group  receiving  training  is  significantly 
higher  than  that  for  the  control  group.  COIC  B  will  be  met  by  SPEARS  if  the  Pfa  exhibited  for  the 
group  receiving  training  is  lower  than  or  not  significantly  different  from  that  of  the  control  group. 
There  are  significant  positive  differences  between  the  training  group  and  the  control  group  for  d. 

3.2  SPEARS  VIGILANCE 


The  results  from  the  ANOVA  carried  out  on  FTI  across  weeks  of  the  study  will  be  reported, 
along  with  associated  mean  and  standard  deviation  for  each  week.  The  results  of  the  post  hoc 
comparisons  conducted  on  any  significant  main  effect  will  also  be  reported.  COIC  A  will  be  met 
by  SPEARS  if  the  Pd  exhibited  for  the  training  group  is  significantly  higher  at  the  midpoint  of  the 
study  than  at  the  beginning.  COIC  B  will  be  met  by  SPEARS  if  the  Pd  exhibited  for  the  training 
group  is  not  significantly  lower  at  the  end  of  the  study  than  at  the  midpoint 

3.3  SPEARS  USABILITY  ASSESSMENT. 


Hardware,  software,  or  procedural  deficiencies  noted  will  be  reported  in  tabular  fonn,  along  with 
any  HF  engineering  recommendations  for  rectification. 

3.4  INSERTION. 


Results  will  be  reported,  including  significant  effects  and  associated  means  and  standard 
deviations.  Any  average  rating  of  “less  than  likely  to  encounter”,  as  measured  by  the  Insertion 
Representativeness  Assessment  form  (see  Appendix  CC),  will  be  deemed  to  be  a  deficiency  and 
the  criterion  will  not  have  been  met. 

4  TECHNICAL  TEST  AND  EVALUATION. 

4.1  TECHNICAL  TEST  PROCEDURES. 


This  section  describes  the  manner  in  which  data  will  be  gathered  to  assess  the  technical 
compliance  of  the  SPEARS  TIP  component.  The  technical  testing  protocols  will  consist  of  HF 
engineers  evaluating  the  SPEARS  device  against  the  technical  criteria  using  checklists  and 
subjective  ratings  by  screeners  and  supervisors. 
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41.1  Image  Content. 


Researchers  will  use  a  SPEARS  Image  Content  Checklist  (see  Appendix  D)  to  determine  whether 
the  technical  criteria  have  been  met  by  posing  the  following  questions  and  documenting  the 
answers  accordingly; 

a.  Are  300  different  CTIs  and  300  FTIs  available? 

b.  Do  the  CTIs  and  FTIs  represent  the  required  range  of  threat  categories  of 
weapons,  incendiaries,  explosives,  and  other  dangerous/deadly  devices? 

c.  What  is  the  distribution  of  the  300  CTIs  and  300  FTIs  by  threat  category? 

d.  Are  two  different  aspect  angles  available  for  each  CTI  and  each  FTI? 

4.1.2  Customization. 


The  following  research  questions  will  be  addressed  during  the  course  of  the  technical  T&E  and 
the  results  will  be  documented  in  a  SPEARS  Customization  Checklist  (Appendk  E)  accordingly: 

a.  Can  CTI  and  FTIs  be  selected  for  each  screener  by  threat  category? 

b.  Can  CTIs  and  FTIs  be  selected  for  each  screener  by  difficulty?  For  the  purposes 
of  this  test,  assessments  of  difficulty  will  be  provided  by  the  vendor. 

c.  Can  CTIs  and  FTIs  be  selected  for  each  screener  by  aspect  angle? 

d.  Can  CTIs  and  FTIs  be  selected  as  a  function  or  combination  of  categories  a. 
through  c.? 

e.  Can  CTIs  and  FTIs  be  archived  along  with  the  associated  outcome  (hit,  miss,  &lse 
alarm,  correct  rejection)  for  each  screener,  along  with  the  date  and  time  of  trial? 

f  Can  CTIs  and  FTIs  be  automatically  or  manually  selected  based  on  time  of  day? 

g.  Can  CTIs  and  FTIs  be  automatically  or  manually  selected  based  on  checkpoint 
activity? 

h.  Can  CTIs  and  FTIs  be  automatically  or  manually  selected  based  on  screener 
identity? 

i.  Can  CTIs  and  FTIs  be  selected  as  a  function  or  combination  of  categories  f 
through  h.? 

4.1.3  Feedback. 


A  HF  engineer  will  answer  the  following  questions  using  a  SPEARS  Feedback  Checklist 
(Appendix  F): 

a.  Does  the  SPEARS  device  provide  adequate  feedback  for  correct  identification  of  a 

FTI? 

b.  Does  the  SPEARS  device  remove  the  FTI  from  the  display  to  allow  for  further 
image  scanning  by  the  X-ray  screener  once  the  TIP  key  has  been  selected? 
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c.  Does  the  SPEARS  device  provide  feedback  about  a  missed  FTI? 

d.  Does  the  SPEARS  device  identify  the  location  of  a  missed  FTI? 

e.  Does  the  SPEARS  device  provide  feedback  for  items  that  are  identified  as  having 
threats,  but  do  not  contain  false  threat  images? 

f  What  is  the  nature  of  the  feedback  for  each  of  the  situations  listed  above? 
g.  What  is  the  time  delay  for  feedback  for  each  of  the  situations  listed  above? 

4.1.4  Capability  Summaries. 


A  HF  engineer  will  answer  the  following  questions  using  a  Capabilities  Summaries  Checklist 
(Appendix  G): 


a. 

b. 

c. 

outcomes? 

d. 

e. 
f 
g- 


Does  the  SPEARS  device  automatically  prepare  summary  reports? 

Do  the  summary  reports  contain  descriptive  statistics? 

Can  the  descriptive  statistics  be  categorized  according  to  the  threats  presented  and 

Can  the  descriptive  statistics  be  categorized  by  screener? 

Can  the  descriptive  statistics  be  categorized  according  to  checkpoint  activity  level? 
Can  the  descriptive  statistics  be  categorized  according  to  the  time  of  day? 

Are  the  summary  reports  readily  understandable  by  the  users? 


NOTE:  To  assess  readily  imderstandable  interpretations,  HF  checklists  addressing 
information  presentation  principles  will  be  used  by  HF  engineers.  Furthermore,  HF 
engineers  will  conduct  structured  interviews  with  screeners  and  supervisors 
regarding  the  reports.  The  questions  for  the  structured  interviews  are  as  follows: 


1 .  Do  you  understand  the  purpose  of  the  SPEARS  TIP? 

2.  Given  the  intent  of  the  system,  do  you  understand  the  outputs  fi’om  the 

system? 

3.  How  could  you  use  these  outputs  to  make  changes  or  improve  the 
screener’s  performance? 

4.  Are  there  items  on  the  reports  that  you  are  imsure  about? 

5.  Is  there  information  missing  on  the  reports  that  could  help  you  perform 
your  job  better? 

6.  Is  the  report  format  acceptable  to  you? 

7.  Are  the  graphics  and  numbers  useful,  or  would  you  prefer  a  text 
explanation  of  the  findings? 

8.  Are  there  any  additional  comments  about  the  reports  you  would  like  to 

share? 

h.  Can  the  SPEARS  device  actively  track  the  performance  level  of  an  individual 
screener  within  a  given  screening  session? 


NOTE:  Performance  level  for  this  test  is  defined  as  Pd,  Pf,  and  d'. 


32 


i.  Can  the  SPEARS  device,  when  queried,  notify  a  remotely  situated  supervisor  of 
performance  levels? 

j.  Does  the  SPEARS  device  automatically  notify  a  remotely  situated  supervisor  of 
performance  levels  which  exceed  or  fall  below  supervisor  set  criteria  thresholds? 

k.  Is  this  notification  of  criteria  thresholds  adequate  in  terms  of  alerting  or  required 
response? 

41.5  Interoperability. 


A  telephone  connection  will  be  established  through  modems  between  the  SPEARS  at  the  remote 
location  and  the  SPEARS  at  the  test  site.  This  telephone  connection  will  require  access  to  the 
local  telephone  system  for  remote  connection  and,  possibly,  long  distance  connection.  Using  a 
second  voice  telephone  connection,  two  researchers  will  attempt  to  download  new  images  to  the 
SPEARS  devices.  Additionally,  a  single  researcher  will  attempt  to  access  reports  from  the 
remotely  situated  SPEARS  device.  A  HF  engineer  will  answer  the  following  questions  using  an 
InteroperabUity  Checklist  (Appendbc  H); 

a.  Can  a  device-to-device  or  a  device-to-computer  connection  be  accomplished  over 
standard  telephone  communication  lines,  using  two  people? 

b.  Can  this  connection  be  used  to  download  new  CTIs  and  FTIs  to  the  SPEARS 

device? 

c.  What  is  the  time  required  to  transfer  a  given  CTI  or  FTI  between  remotely  situated 
devices? 

d.  Can  a  device-to-device  or  device-to-computer  connection  be  initiated  remotely  for 
the  collection  of  status  reports? 

e.  Can  the  remote  connection  access  the  report  data  and  save  these  data  to  a  local 
computer  file? 

f  Can  the  remote  connection  subsequently  print  out  the  status  reports? 
g.  Are  the  status  reports  that  are  remotely  available  similar  to  those  available  on-site 
from  the  SPEARS  device? 

4.1.6  Security. 


The  Security  issue  will  be  investigated  using  a  structured  Security  Access  Control  Checklist  (see 
Appendbc  I).  The  security  checklist  is  based  on  the  Department  of  Defense  Trusted  Computer 
System  Evaluation  Criteria  (DOD-5200.28-STD)  and  is  contained  in  Appendbc  G. 

The  SPEARS  computer  software  will  be  exercised  to  determine  if  adequate  security  is  provided. 
The  following  questions  will  be  answered  in  addressing  this  issue; 

a.  Can  unauthorized  personnel  access  the  image  library  areas  of  the  SPEARS? 

b.  Can  unauthorized  personnel  access  the  stored  personnel  records  in  the  SPEARS? 
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4.1.7  Insertion. 


Examination  of  the  insertion  issue  will  occur  using  a  structured  Image  Insertion  Checklist  (see 
Appendix  J). 

Baggage  not  containing  threat  objects  will  be  passed  through  the  X-ray  device,  and  the  SPEARS 
device  will  be  used  to  insert  threat  objects  onto  the  display  of  the  bag  image.  In  addition,  a  series 
of  neutral  bags  Avill  be  run  through  the  X-ray  device,  and  additional  CTIs  will  be  inserted  into  the 
baggage  flow.  A  HF  engineer  will  answer  the  following  questions; 

a.  Can  an  FTI  be  generated  using  a  computer-generated  threat  object  and  a  neutral 
passenger  bag  passed  through  an  operational  X-ray  device  connected  to  a  SPE.^5  device? 

b.  Are  the  FTIs  generated  automatically  by  the  SPEARS  device? 

c.  Are  the  FTIs  predictable  in  the  random  order  that  they  are  presented? 

NOTE.  For  this  test,  based  on  data  from  the  serial  presentation  of  FTIs,  a 
computer  algorithm  will  be  used  to  determine  presentation  order  randomness. 

d.  Can  a  CTI  be  inserted  into  the  flow  of  normal  bags  passed  through  an  X-ray 
device  connected  to  a  SPEARS  device? 

e.  Are  the  CTIs  generated  automatically  by  the  SPEARS  device? 

£  Are  the  CTIs  predictable  in  the  random  order  that  they  are  presented? 

NOTE:  For  this  test,  based  on  data  from  the  serial  presentation  of  CTIs,  a 
computer  algorithm  will  be  used  to  determine  randomization  of  the  presentation 
order. 

g.  Is  the  position  of  the  FTI  randomly  determined? 

NOTE:  For  this  test,  a  series  of  200  neutral  bags  will  be  passed  through  an 
operational  X-ray  device  attached  to  a  SPEARS  device.  FTIs  will  be  projected 
onto  each  neutral  bag  and  the  resulting  image  will  be  stored  for  analysis.  During 
analysis,  each  resultant  image  will  be  examined  by  dividing  the  bag  image  into  nine 
sectors  and  categorizing  where  the  FTI  has  been  placed.  The  matrix  of  bag,  bag 
order  number,  and  corresponding  FTI  position  a^I  be  entered  into  a  statistic^ 
analysis  package  and  analyzed  for  random  position  assignment. 

h.  Is  the  orientation  of  the  FTI  within  each  bag  controllable? 

NOTE:  For  this  test,  an  insertion  order  will  be  established  in  the  SPEARS  device, 
with  orientations  pre-assigned.  A  run  of  test  bags  will  then  be  initiated  iising 
neutral  bags  and  FTIs  imposed.  The  run  will  then  be  stored  and  re-examined  to 
check  for  concordance  between  the  pre-assigned  orientation  and  the  orientations 
produced  by  the  SPEARS  device. 
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4.2  DATA. 


The  data  resulting  from  the  technical  MOPs  include  the  answers  to  the  structured  interview 
question  concerning  summary  report  understandability,  annotated  results  from  the  review  of 
pertinent  aspects  of  the  SPEARS  device  against  HFE  checklists,  assessments  of  FTI  and  CTI 
insertion  order  and  FTI  position  assignment  within  bag  images,  image  control,  and  researcher 
observations. 

4.3  TEST  DATA  MANAGEMENT. 


Data  coUected  during  the  test  will  be  transferred  into  electronic  media  and  stored  after  the  test  of 
the  SPEARS  TIP.  Original  hand-written  data  and  research  observations  will  be  maintained  in 
Galaxy  Scientific  Corporation  (GSC)  files  and  will  be  made  available  to  the  FAA  upon  request. 

4.4  DATA  ANALYSIS. 


MOP  data  for  each  technical  issue  will  be  assessed  against  its  related  criteria  to  ensure  compliance 
with  system  technical  requirements.  The  individual  SPEARS  technical  issues  checklists  will  be 
assessed  to  ascertain  the  extent  these  criteria  have  been  met.  Responses  from  the  structured 
interviews  will  augment  the  lED  test,  TIP  performance,  and  technical  issue  checklist  data  to 
enhance  the  comprehension  of  the  results.  Although  the  structured  interview  data  are  subjective, 
they  will  be  evaluated  relative  to  the  extent  that  it  supports  the  objective  performance  data. 

5.  PRESENTATION  OF  TECHNICAL  TEST  AND  EVALUATION  RESULTS. 

The  results  of  the  technical  issues  investigations  will  be  presented  in  terms  of  the  technical  criteria 
and  associated  MOPs  developed  previously.  Technical  test  results  will  be  categorized  by 
technical  issues  and  reported  accordingly.  System  deficiencies  evidenced  by  the  data  collected 
will  be  identified  as  an  aspect  of  the  SPEARS  that  did  not  meet  the  technical  criteria.  In  this 
event,  recommendations  for  resolution  will  be  made. 
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Appendix  A 
to  SPEARS  OT&ETEP 


SIGNAL  DETECTION  THEORY  AND  APPLICATION 
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SIGNAL  DETECTION  THEORY  AND  APPLICATION 


SIGNAL  DETECTION  THEORY  fSDT^  PARADIGM. 

The  Improvised  Explosive  Device  Detection  System  operation  features  human  operators  engaged 
in  tasks  to  detect  an  environmental  event  or  signal.  SDT  is  a  mathematical  representation  of 
human  performance  in  deciding  whether  or  not  a  signal  is  present.  An  operational  example  of 
SDT  is  an  airport  security  guard  screening  passenger  bags  for  concealed  weapons  and  Improvised 
Explosive  Devices  (lEDs). 

There  are  two  response  categories  that  represent  a  screener’s  detection  performance:  Yes  (a 
Modular  Bomb  Set  [MBS]  signal  was  present)  or  No  (an  MBS  signal  was  not  present).  There  are 
also  two  signal  presentation  states  indicating  that  the  MBS  signal  was  present  (signal)  or  absent 
(noise).  A  combination  of  screener  responses  and  the  signal  state  produces  a  2  x  2  matrix  (figure 
A-1),  generating  four  classes  of  operator  responses,  labeled  hits,  misses,  false  alarms,  and  correct 
rejections  (Wickens  1992).  Considering  the  Screener  Proficiency  Training  Evaluation  and 
Reporting  System  (SPEARS)  Test  and  Evaluation: 

State  of  MBS  Image 

MBS  Present  MBS  Not  Present 


Yes 


Screener 

Response 

No 


Hit 

False  Alarm 

Miss 

Correct 

Rejection 

FIGURE  A-1.  2  X  2  MATRIX  OF  SCREENER  RESPONSES 
AND  STATE  OF  MBS  IMAGE 

a.  A  Hit  will  be  recorded  when  a  baggage  screener  correctly  detects  an  MBS  in  the 
scanned  baggage. 

b.  A  False  Alarm  will  be  recorded  when  a  baggage  screener  detects  an  MBS  in  the 
scanned  baggage  when  none  is  present. 
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As  indicated  by  Wickens  (1992),  the  SDT  paradigm  assumes  that  operators  perform  two  stages 
of  information  processing  in  all  detection  tasks:  (1)  sensory  evidence  is  aggregated  concerning 
the  presence  or  absence  of  the  signal,  and  (2)  a  decision  is  made  about  whether  this  evidence 
constitutes  a  signal.  According  to  SDT,  external  stimuli  generate  neural  activity  in  the  brain.  On 
the  average,  there  will  be  more  sensory  or  neural  evidence  in  the  brain  when  a  signal  is  present 
than  when  it  is  absent.  This  neural  evidence,  X,  referred  to  as  the  evidence  variable,  represents 
the  rate  of  firing  of  neurons  in  the  brain.  The  response  rate  for  detecting  X  increases  in 
magnitude  with  stimulus  (signal)  intensity.  Therefore,  if  there  is  enough  neural  activity,  X 
exceeds  a  critical  threshold,  X^,  and  the  operator  decides  “yes.”  If  there  is  too  little,  the  operator 

decides  “no.”  Because  the  amount  of  energy  in  the  signal  is  typically  low,  the  average  amount  of 
X  generated  by  signals  in  the  environment  is  not  much  greater  than  the  average  generated  when 
no  signals  are  present  (noise).  Furthermore,  the  quantity  of  X  varies  continuously,  even  in  the 
absence  of  a  signal,  because  of  random  variations  in  the  environment  and  the  operator’s  level  of 
neural  firing  (i.e.,  the  neural  “noise”  in  the  operator’s  sensory  channels  and  brain). 

The  relationship  between  the  presence  and  absence  of  a  signal  can  be  seen  in  the  hypothetical 
noise  and  signal  plus  noise  distributions  contained  in  figure  A-2.  The  intersection  of  the  two 
curves  represents  the  location  where  the  probability  of  a  signal  equals  the  probability  of  noise. 
The  criterion  value,  X^,  chosen  by  the  operator,  is  shown  by  the  vertical  line.  All  X  values  to  the 

right  (X  >  X^)  will  cause  the  operator  to  respond  “yes,”  All  X  values  to  the  left  generate  “no” 
responses. 


FIGURE  A-2.  HYPOTHETICAL  SDT  DISTRIBUTIONS  (Wickens  1992) 


The  different  shaded  areas  represent  the  occurrences  of  hits,  misses,  false  alarms,  and  correct 
rejections. 
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The  procedures  to  calculate  SDT  probabilities  are  as  follows: 

a.  In  SDT,  the  detection  values  are  expressed  as  probabilities. 

b.  The  probability  of  hit  (Ph),  miss  (P^^),  false  alarm  and  correct  rejection  (P^^) 

are  determined  by  dividing  the  number  of  occurrences  in  a  cell  (figure  A-1)  by  the  total  number 
of  occurrences  in  a  column. 

c.  The  P|j  (also  referred  as  the  probability  of  detection  [P^])  will  be  calculated  by 
dividing  the  number  of  lEDs  detected  (number  of  hits)  by  the  total  number  of  hits  and  misses: 

m  d 

d.  The  Pf^  will  be  determined  by  the  number  of  false  alarms  divided  by  the  total 
number  of  false  alarms  and  correct  rejections:  P^  =  1  -  P^^. 

Operator  Response  Criterion. 

In  any  signal  detection  task,  operator  decision  making  may  be  described  in  terms  of  an  operator 
response  criterion.  Operators  may  use  “risky”  response  strategies  by  responding  yes  more  often 
than  no.  A  risky  strategy  allows  operators  to  detect  most  of  the  signals  that  occur,  but  also 
produces  many  false  alarms.  Alternatively,  operators  may  use  “conservative”  strategies,  saying 
no  most  of  the  time,  making  few  false  alarms,  but  missing  many  of  the  signals. 

Different  circumstances  may  require  conservative  or  risky  strategies.  For  example,  an 
appropriate  lED  detection  strategy  requires  screeners  to  respond  “yes”  when  there  is  a  question 
regarding  baggage  contents.  This  response  may  produce  false  alarms  when  no  threatening 
objects  are  present. 

As  shown  in  figure  A-2,  risky  or  conservative  behavior  is  determined  by  the  location  of  the 
operator’s  response  criterion,  X^.  If  is  placed  to  the  right,  much  evidence  of  the  signal  is 

required  for  it  to  be  exceeded  and  most  responses  will  be  “no”  (conservative  responding).  If  it  is 
placed  to  the  left,  little  signal  evidence  is  required  and  most  responses  will  be  “yes,”  or  “risky.” 
A  variable  positively  correlated  with  X^  is  the  quantity  beta  (P),  which  is  determined  as  follows: 


ORDh 

ORDfa 


(1) 


This  equation  is  the  ratio  of  the  ordinate  of  the  two  curves  of  figure  A-2  at  a  given  level  of  X^. 

The  higher  p  values  will  generate  fewer  yes  responses  and,  therefore,  fewer  hits.  Lower  p 
settings  will  generate  more  yes  responses,  more  hits,  and  more  false  alarms. 
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Table  A-1  provides  a  representative  table  of  Z  values  and  ordinate  values  of  the  probability 
distribution  related  to  hit  and  false  alarm  responses.  A  complete  table  of  standard  normal 
distribution  values  will  be  used  to  calculate  p  for  the  Test  and  Evaluation  Report  (TER). 


TABLE  A-1.  REPRESENTATIVE  Z-SCORES  AND  ORDINATE  VALUES  OF  THE 
NORMAL  CURVE  FOR  DIFFERENT  RESPONSE  PROBABILITIES 
TO  CALCULATE  p  AND  d' 


HIT/FA 

Z 

ORD 

HIT/FA 

Z 

ORD 

.01 

2.33 

0.03 

.50 

0.00 

0.40 

.02 

2.05 

0.05 

.55 

-0.12 

0.40 

.03 

1.88 

0.07 

.60 

-0.25 

0.39 

.04 

1.75 

0.09 

.65 

-0.38 

0.37 

.05 

1.64 

0.10 

.70 

-0.52 

0.35 

.08 

1.40 

0.15 

.75 

-0.67 

0.32 

.10 

1.28 

0.18 

.80 

-0.84 

0.28 

.13 

1.13 

0.21 

.82 

-0.92 

0.26 

.15 

1.04 

0.23 

.85 

-1.40 

0.23 

.18 

0.92 

0.26 

.88 

-1.18 

0.20 

.20 

0.84 

0.28 

.90 

-1.28 

0.18 

.25 

0.67 

0.32 

.92 

-1.40 

0.15 

.30 

0.52 

0.35 

.95 

-1.64 

0.10 

.35 

0.38 

0.37 

.96 

-1.75 

0.09 

.40 

0.25 

0.39 

.97 

-1.88 

0.07 

.45 

0.12 

0.40 

.98 

-2.05 

0.05 

.50 

0.00 

0.40 

.99 

-2.33 

0.03 

The  procedures  required  to  calculate  p  are  listed  below  (Coren  and  Ward  1989): 

a.  Find  the  false  alarm  rate  from  the  outcome  matrix  in  the  HIT/FA  column  of  table 

A-1. 

b.  Read  across  the  table  to  the  ORD  column  (for  ordinate,  the  height  of  the  bell 

curve). 

c.  Determine  the  value  tabled  there  and  write  it  down. 

d.  Repeat  these  operations  for  the  hit  rate. 

e.  Calculate  P  using  the  following  equation:  P  =  ORD^/ORD^^. 
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One  recent  parametric  measure  of  response  bias  is  c  (Ingham  1970;  Macmillan  &  Creelman 
1990;  and  Snodgrass  &  Corwin  1988).  The  chief  difference  between  the  measure  c  and  its 
parametric  alternative  p  lies  in  the  manner  in  which  they  locate  the  observer’s  criterion.  Whereas 
the  bias  index  p  locates  the  observer’s  criterion  by  the  ratio  of  the  ordinates  of  the  signal-plus  - 
noise  (SN)  and  noise  (N)  distributions,  c  locates  the  criterion  by  its  distance  from  the  intersection 
of  the  two  distributions  measured  in  z-score  units.  The  intersection  defines  the  point  where  bias 
is  neutral,  and  location  of  the  criterion  at  that  point  yields  a  c  value  of  0.  Conservative  criteria 
yield  positive  c  values,  and  liberal  criteria  produce  negative  c  values.  The  measure  c  is 
computed  as  follows: 


C  =  .5(Zfa  +  Zh) 


(2) 


Sensitivity  fd^V 

Sensitivity  refers  to  the  average  amount  of  operator  sensory  activity  generated  by  a  given  signal 
as  compared  with  the  average  amount  of  noise-generated  activity  (Coren  and  Ward  1989).  As 
explained  earlier,  baggage  screeners  may  fail  to  detect  (miss)  an  lED  signal  when  employing  a 
conservative  p.  Correspondingly,  the  signal  may  be  missed  because  the  resolution  of  the 
detection  process  is  low  in  discriminating  signals  from  noise,  even  if  p  is  neutral  or  risky. 

The  perceptual  analog  of  sensitivity,  d',  corresponds  to  the  separation  of  the  means  of  signal  and 
noise  distributions  (figure  A-2),  As  the  magnitude  of  the  signal  increases,  the  mean  of  the  signal 
distribution  moves  to  the  right.  The  proportion  of  signals  detected  (the  P^)  changes  as  the 
distance  between  the  signal  and  noise  distributions  varies.  According  to  Wickens  (1992),  if  the 
separation  between  the  distributions  is  great,  sensitivity  is  great,  an  operator  can  readily 
distinguish  a  signal  plus  noise  event  from  a  noise  only  event.  Similarly,  if  the  separation  between 
signal  and  noise  is  small,  d'  measures  will  be  low. 

Table  A-1  provides  a  representative  table  of  Z  values  and  ordinate  values  for  the  probability 
distribution  related  to  hit  and  false  alarm  responses.  A  complete  table  of  the  ordinate  values  for 
the  standard  normal  distribution  will  be  used  to  calculate  d'  for  the  TER. 

The  procedures  required  to  calculate  d'  are  listed  below  (McNicol  1972). 

a.  Separate  the  data  with  respect  to  distribution  (signal  or  noise),  and  total  the  data 
with  respect  to  response  (yes  or  no)  and  confidence  rating  (very  sure  or  not  very  sure). 

b.  Calculate  and  Pfa  category. 

c.  Convert  the  values  of  P^  and  ^ fa  to  Zd  and  Zfa  using  the  normal  curve  area. 

d.  Plot  Zd  against  Zfa  on  a  double-probability  plot  and  determine  the  path  of  the  ROC 
curve  based  upon  the  method  of  maximum  likelihood. 
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e.  Determine  the  ordinate  value,  for  either  axis,  at  which  the  ROC  curve  intersects 
with  the  negative  diagonal.  This  value  is  equal  to  I6d'.  Multiply  this  value  by  2  for  d  . 
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Appendix  B 
to  SPEARS  OT&E  TEP 


SPEARS  USABILITY  HFE  CHECKLIST 
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SPEARS  USABILITY  HFE  CHECKLIST 


Human  Factors  Principle 


1.0  Data  Entry 


1.  Users  need  enter  data  only  once. 


2.  Display  feedback  for  all  user  actions 
during  data  entry;  display  keyed  entries 
stroke  by  stroke. 


3.  Provide  fast  response  by  the  computer 
in  acknowledging  data  entries. 


4.  Incorporate  a  consistent  method  for 
data  change. 


5.  When  critical  data  are  to  be  processed, 
require  an  explicit  “Enter”  action  to 
initiate  the  processing. 


6.  Provide  feedback  for  the  completion 
of  data  entry. 


7.  For  repetitive  data  entry  transactions, 
feedback  should  be  the  generation  of  or 
moving  to  the  next  data  entry  field. 


8.  Choose  defining  abbreviations  or 
other  codes  to  shorten  data  entry  that  are 
distinctive  to  avoid  confusion  with  one 
another. 

9.  Allow  users  to  enter  each  character  of 
a  data  item  with  a  single  stroke  of  an 
appropriately  labeled  key. 

10.  Treat  upper  and  lower  case  letters  as 
equivalent. 
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39.  Include  in  a  field  label  additional 
cueing  of  data  format  when  that  seems 
helpful. _ 
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Human  Factors  Principle 


40.  When  a  measurement  unit  is 
consistently  associated  with  a  particular 
data  field,  include  that  unit  as  part  of  the 
field  label,  rather  than  requiring  a  user  to 
enter  it. 


Evaluation 


Acceptable  with 
Deficiencies 


Comments 


41.  Employ  units  of  measurement  that 
are  familiar  to  the  user. 


42.  Order  data  items  in  the  sequence  in 
which  a  user  will  think  of  them. 


43,  The  order  of  data  items  should 
represent  a  logical  sequence  of  data 
entries. 


44.  When  a  form  for  data  entry  is 
displayed,  the  computer  should  place  the 
cursor  automatically  at  the  beginning  of 
the  first  entry  field. 


45.  When  sets  of  data  items  must  be 
entered  sequentially  in  a  repetitive  series, 
provide  a  tabular  display  format  where 
data  sets  can  be  keyed  row  by  row. 


46.  Design  distinctive  formats  for 
column  headers  and  row  labels,  so  that 
users  can  distinguish  them  from  data 
entries. 


47.  Ensure  that  column  headers  and  row 
labels  are  worded  informatively,  so  that 
they  will  help  guide  data  entry. 


48.  During  tabular  data  entry,  allow  users 
to  tab  directly  from  one  data  field  to  the 
next,  so  that  the  cursor  can  move  freely 
back  and  forth. 
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_ Human  Factors  Principle _ 

49.  When  a  user  must  enter  numeric 
values  that  will  later  be  displayed, 
maintain  all  signiflcant  zeros;  zeros 
should  not  be  arbitrarily  removed  after  a 
decimal  point  if  they  affect  the  meaning 
of  the  number  in  terms  of  significant 
digits. 


50,  For  entry  of  tabular  data,  when 
entries  are  frequently  repeated,  provide 
users  with  some  easy  means  to  copy 
duplicated  data. 

51.  Provide  users  some  means  for 
designating  and  selecting  displayed 
graphic  elements  for  manipulation. 
When  a  user  has  changed  or  altered  a 
displayed  element,  provide  some 
indication  so  that  other  users  will  be 
aware  of  the  consequences  of  that  users 
actions. 


53.  Ensure  that  every  possible  correct 
data  entry  will  be  accepted  and 
processed  properly  by  the  computer. 

54.  If  automatic  data  validation  detects  a 
probable  error,  display  an  error  message 
to  the  user.  If  data  validation  from  a 
secondary  source  detects  a  probable 
error,  allow  the  primary  user  access  to  all 
functions  required  to  check  data 
accuracy. 

55.  If  a  user  has  deferred  entry  of 
required  data,  but  then  requests 
processing  of  entries,  signal  that 
omission  to  the  user  and  allow  immediate 
entry  of  missing  items  or  perhaps  further 
deferral. 


56.  If  a  set  of  default  values  have  been 
defined  for  a  data  entry  sequence,  allow 
a  user  to  accept  all  default  entries. 
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_ Human  Factors  Principle _ 

5.  For  any  particular  type  of  data  display, 
maintain  consistent  format  from  one 
display  to  another. 

6.  Allow  users  to  change  displayed  data 
or  enter  new  data  when  that  is  required 
by  a  task. 

7.  When  protection  of  displayed  data  is 
essential,  maintain  computer  control  over 
the  display  and  do  not  permit  a  user  to 
change  controlled  items. 


8.  Ensure  that  each  data  display  will 
provide  needed  context,  recapitulating 
prior  data  as  necessary  so  that  a  user 
does  not  have  to  rely  on  memory  to 
interpret  new  data. 

9.  The  wording  of  displayed  data  and 
labels  should  incorporate  familiar  terms 
and  the  task-oriented  jargon  of  the  users. 

10.  Choose  words  carefully  and  then  use 
them  consistently. 


11.  Ensure  that  wording  is  consistent 
from  one  display  to  another. 

12.  Use  consistent  grammatical  structure 
within  and  across  displays. 

13.  When  abbreviations  are  used,  choose 
those  abbreviations  that  are  commonly 
recognized,  and  do  not  abbreviate  words 
that  produce  uncommon  or  ambiguous 
abbreviations. 


14.  Ensure  that  abbreviations  are 
distinctive,  so  that  abbreviations  for 
different  words  are  distinguishable. 
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Human  Factors  Principle 


24.  In  designing  text  displays,  especially 
text  composed  for  user  guidance,  strive 
for  simplicity  and  clarity  of  wording. 


Evaluation 


Acceptable  with 
Deficiencies 


Comments 


25.  Put  the  main  topic  of  each  sentence 
near  the  beginning  of  the  sentence. 


26.  Use  short  simple  sentences. 


27.  Use  affirmative  statements  rather 
than  negative  statements. 


28.  Compose  sentences  in  the  active 
voice  rather  than  passive  voice. 


29.  When  a  sentence  describes  a 
sequence  of  events,  phrase  it  with  a 
corresponding  word  order. 


30.  Format  lists  so  that  each  item  starts 
on  a  new  line  (i.e.,  a  list  should  be 
displayed  as  a  single  column). 


31.  When  a  single  item  in  a  list  continues 
for  more  than  one  line,  mark  items  in 
some  way  so  that  the  continuation  of  an 
item  is  obvious  (i.e.,  so  that  a  continued 
portion  docs  not  appear  to  be  a  separate 
item). 


32.  When  listed  items  will  be  numbered, 
use  Arabic  rather  than  Roman  numerals. 


33.  Adopt  some  logical  principle  by 
which  to  order  lists.  When  no  other 
principle  applies,  order  lists 
alphabetically. 


34.  If  a  list  is  displayed  in  multiple 
columns,  order  the  items  vertically 
within  each  cell. 
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Human  Factors  Principle 


44.  Use  consistent  logic  in  the  design  of 
graphic  displays,  and  maintain  standard 
format,  labeling,  etc. 


Evaluation 


Acceptable  with 
Deficiencies 


Comments 


45.  Tailor  graphic  displays  to  user  needs 
and  provide  only  those  data  necessary  for 
user  tasks. 


46.  When  graphics  contain  outstanding 
or  discrepant  features  that  merit  attention 
by  a  user,  consider  displaying 
supplementary  text  to  emphasize  that 
feature. 


47.  When  a  user’s  attention  must  be 
directed  to  a  portion  of  a  display 
showing  critical  or  abnormal  data, 
highlight  that  feature  with  some 
distinctive  means  of  coding. 


48.  Adopt  a  consistent  organization  for 
the  location  of  various  display  features 
from  one  display  to  another. 


49.  Make  the  different  elements  of  a 
display  format  distinctive  from  one 
another. 


50.  Use  blank  space  to  structure  a 
display. 


51.  Begin  every  display  at  the  top  with  a 
title  or  header,  describing  briefly  the 
contents  or  purpose  of  the  display. 


52.  Reserve  the  last  several  lines  at  the 
bottom  of  every  display  for  status  and 
error  messages,  prompts,  and  command 
entry. 


53.  Ensure  that  displays  are  formatted  to 
group  data  items  on  the  basis  of  some 
logical  principle. 


_ Human  Factors  Principle _ 

54.  Provide  distinctive  coding  to 
highlight  important  display  items 
requiring  user  attention.  Highlighting  is 
most  effective  when  used  sparingly. 

55.  If  highlighting  is  used  to  emphasize 
important  display  items,  remove  such 
highlighting  when  it  no  longer  has 
meaning. 

56.  Adopt  meaningful  or  familiar  codes, 
rather  than  arbitrary  codes. 

57.  Adopt  codes  for  display  (and  entry) 
that  conform  with  accepted  abbreviations 
and  general  user  expectations. 

58.  Assign  consistent  meanings  to 
symbols  and  other  codes,  from  one 
display  to  another. 

59.  Treat  brightness  as  a  two-valued 
code,  bright  and  dim,  i.e.,  consider 
coding  by  difference  in  brightness  for 
applications  that  only  require 
discriminating  between  two  categories  of 
displayed  items. 

60.  Color  code  for  relative  values. 


61.  It  will  prove  safer  to  use  no  more 
than  five  different  colors  for  coding. 

62.  When  selecting  colors  for  coding 
discrete  categories  of  data,  ensure  that 
those  colors  are  easily  discriminable. 

63.  Employ  color  coding  conservatively, 

using  relatively  few  colors  and  only  to 
designated  critical  categories  of 
displayed  data. _ 
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Human  Factors  Principle 


2.  Employ  similar  means  to  accomplish 
ends  that  are  similar,  from  one 
transaction  to  the  next,  from  one  task  to 
another,  throughout  the  user  interface. 


Evaluation 


Acceptable  with 
Deficiencies 


Comments 


3.  Display  some  continuous  indication  of 
current  context  for  reference  by  the  user. 


4.  Adopt  consistent  terminology  for  on¬ 
line  guidance  and  other  messages  to 


5.  Choose  names  that  are  semantically 
congruent  with  natural  usage,  especially 
for  paired  opposites  (e.g.,  UP  /  DOWN). 


6.  Ensure  that  the  computer 
acknowledges  every  entry  immediately; 
for  every  action  by  the  user  there  should 
be  some  apparent  reaction  from  the 
computer. 


7.  Each  menu  display  should  permit  only 
one  selection  by  the  user. 


8.  When  multiple  menu  options  are 
displayed  in  a  list,  display  each  option  on 
a  new  line,  i.e.,  format  the  list  as  a  single 
column. 


9.  When  a  user  has  selected  and  entered 
an  option  from  a  menu,  if  there  is  no 
immediately  observable  natural  response, 
then  the  computer  should  display  some 
other  acknowledgment  of  that  entry. 


10.  Display  an  explanatory  title  for  each 
menu,  reflecting  the  nature  of  the  choice 
to  be  made. 


11.  Design  a  menu  to  display  all  options 
appropriate  to  any  particular  transaction. 
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Human  Factors  Principle 


20.  When  hierarchic  menus  are  used, 
ensure  that  display  format  and  selection 
logic  are  consistent  at  every  level. 


Evaluation 


Acceptable  with 
Deficiencies 


Comments 


21,  When  hierarchic  menus  are  used, 
require  users  to  take  only  one  simple  key 
action  to  return  to  the  next  higher  level. 


22.  When  hierarchic  menus  are  used, 
require  users  to  take  only  one  simple  key 
action  to  return  to  the  general  menu  at 
the  top  level. 


23,  Design  the  general  options  list  to 
show  control  entry  options  grouped, 
labeled,  and  ordered  in  terms  of  their 
logical  function,  and  frequency  and 
criticality  of  use. 


24.  Make  available  to  users  a  list  of  the 
control  options  that  are  specifically 
appropriate  for  any  transaction. 


25.  Offer  users  only  control  options  that 
are  actually  available  for  the  current 
transactions. 


26.  When  a  user  is  performing  an 
operation  on  some  selected  display  item, 
highlight  that  item. 


27.  Design  the  interface  software  to  deal 
appropriately  with  all  possible  control 
entries,  correct  and  incorrect. 


28.  When  a  user  completes  correction  of 
an  error,  require  the  user  to  take  an 
explicit  action  to  reenter  the  corrected 
material;  use  the  same  action  for  reentry 
that  was  used  for  the  original  entry. 
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Human  Factors  Principle 


3.  In  applications  where  users  must  log 
on  to  the  system,  design  LOG-ON  as  a 
separate  procedure  that  is  completed 
before  a  user  is  required  to  select  among 
any  operational  options. 


Evaluation 


Acceptable  with 
Deficiencies 


Comments 


4.  Ensure  that  only  relevant  data  are 
displayed  by  tailoring  the  display  for  any 
transaction  to  the  current  information 
requirements  of  the  user. 


5.  Create  display  formats  with  a 
consistent  structure  evident  to  the  user, 
so  that  a  particular  type  of  data  is  always 
presented  in  the  same  place  and  in  the 
same  way. 


6.  Format  each  different  type  of  user 
guidance  consistently  across  displays. 


7.  Design  display  formats  so  that  user 
guidance  material  is  readily 
distinguishable  from  displayed  data. 


8.  Design  cursors  so  that  they  arc  readily 
distinguished  from  other  displayed  items. 


9.  Label  all  displayed  data  clearly. 


10.  Ensure  that  symbols  and  other  codes 
have  consistent  meanings  from  one 
display  to  another. 


11.  Ensure  that  the  names  for  functions 
and  data  are  consistent  for  similar  or 
identical  functions. 


12.  Adopt  wording  terminology  familiar 
to  users. 


13.  Adopt  task-oriented  wordin 
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Human  Factors  Principle 


14.  Adopt  affirmative  rather  than 
negative  wording  for  user  guidance 
messages. 


Evaluation 


Acceptable  with 
Deficiencies 


Comments 


15.  Adopt  active  rather  than  passive 
voice  in  user  guidance  messages. 


16.  When  user  guidance  describes  a 
sequence  of  steps,  follow  that  same 
sequence  in  the  wording  of  user 
guidance. 


17.  Be  consistent  in  grammatical 
construction  when  wording  user 
guidance. 


18.  Allow  users  to  switch  easily  between 
information  handling  and  associated 
guidance  material. 


19.  Provide  some  indication  of  system 
status  to  users  at  all  times. 


20.  When  users  must  log  on  to  a  system, 
display  appropriate  prompts  for  LOG¬ 
ON  procedures  automatically  at  a  user’s 
terminal. 


21.  If  a  user  tries  to  log  onto  a  system 
and  LOG-ON  is  denied  because  of 
system  unavailability,  display  an 
advisory  message  telling  the  user  what 
the  system  status  is  and  when  the  system 
will  become  available. 


22.  When  task  performance  requires  data 
exchange  and/or  interaction  with  other 
users,  allow  a  user  to  obtain  status 
information  concerning  other  people 
currently  using  the  system. 
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Human  Factors  Principle 


23.  Ensure  that  every  input  by  a  user  will 
consistently  produce  some  perceptible 
response  output  from  the  computer. 


Evaluation 


Acceptable  with 
Deficiencies 


Comments 


24.  Ensure  that  computer  response  to 
user  entries  will  be  rapid,  with  consistent 
timing  as  appropriate  for  different  types 
of  transactions. 


25.  Provide  some  indication  of 
transaction  status  whenever  the  complete 
response  to  a  user  entry  will  be  delayed. 


26.  Provide  a  unique  identification  for 
each  display  in  a  consistent  location  at 
the  top  of  the  display  frame. 


27.  When  a  user  (or  computer)  action 
establishes  a  change  in  operational  mode 
that  will  affect  subsequent  user  actions, 
display  some  continuing  indication  of 
current  mode. 


28.  When  the  computer  detects  an  entry 
error,  display  an  error  message  to  the 
user  stating  what  is  wrong  and  what  can 
be  done  about  it. 


29.  Make  the  wording  of  error  messages 
as  specific  as  possible. 


30.  Make  error  messages  brief  but 
informative. 


31.  Adopt  neutral  wording  for  error 
messages;  do  not  imply  blame  to  the 
user,  or  personalize  the  computer,  or 
attempt  to  make  a  message  humorous. 


32.  The  computer  should  display  an  error 
message  only  after  a  user  has  completed 
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42,  In  addition  to  explicit  and  implicit 
aids,  permit  users  to  obtain  further  on¬ 
line  guidance  by  requesting  HELP. 
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Human  Factors  Principle 


43.  Provide  a  simple,  standard  action  that 
is  always  available  to  request  HELP. 


Evaluation 


Acceptable  with 
Deficiencies 


Comments 


44.  When  an  initial  HELP  display 
provides  only  summary  information, 
provide  more  detailed  explanations  in 
response  to  repeated  user  requests  for 
HELP, 


45.  Permit  users  to  browse  through  on¬ 
line  HELP  displays,  just  as  they  would 
through  a  printed  manual,  to  gain 
familiarity  with  system  functions  and 
operating  procedures. 


5.0  Data  Transmission 


1.  Choose  functional  wording  for  terms 
used  in  data  transmission,  including 
messages,  for  initiating  and  controlling 
message  transmission  and  other  forms  of 
data  transfer,  and  for  receiving  messages. 


2.  Design  the  data  transmission 
procedures  to  minimize  memory  load  on 
the  user. 


3.  Design  the  data  transmission 
procedures  to  minimize  required  user 
actions. 


4.  Design  the  data  transmission 
procedures  so  that  both  sending  and 
receiving  messages  are  accomplished  by 
explicit  user  action. 


5.  Allow  users  to  interrupt  message 
preparation,  review,  or  disposition,  and 
then  resume  any  of  those  tasks  from  the 
oint  of  interruption. 
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Human  Factors  Principle 


6.  Provide  software  capabilities  to 
annotate  transmitted  data  with 
appropriate  highlighting  to  emphasize 
alarm/alert  conditions,  priority 
indicators,  or  other  significant  second- 
order  information  that  could  affect 
message  handling. 


Evaluation 


Acceptable  with 
Deficiencies 


Comments 


7.  For  addressing  and  identifying 
messages,  provide  a  basic  set  of  header 
fields  that  can  be  interpreted  by  all 
systems  to  which  users  will  send 
messages, 

8.  Allow  users  to  print  copies  of 
transmitted  messages  in  order  to  make 
hard-copy  records. 


9.  Ensure  that  transmitted  data  are 
protected  automatically  with  parity 
checks  to  detect  and  correct  any  errors 
that  may  occur,  and  buffering  until 
acknowledgment  of  receipt. 


10.  Ensure  that  only  one  copy  of  a 
message  will  be  transmitted  to  any 
individual  addressee. 


1 1.  For  receiving  messages,  allow  users 
to  choose  the  method  of  receipt  (i.e., 
files,  display,  printer)  that  will  be  the 
local  destination. 


12.  Ensure  that  computer  aids  and 
procedures  for  reviewing  messages  are 
consistent  with  other  system  capabilities 
for  general  data  display. 


6.0  Data  Protection 


1.  Provide  automatic  measures  to 
minimize  data  loss  from  computer 
failure. 
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_ Human  Factors  Principle _ 

2.  Protect  data  from  inadvertent  loss 
caused  by  the  actions  of  other  users. 

3.  Provide  clear  and  consistent 
procedures  for  different  types  of 
transactions,  particularly  those  involving 
data  entry,  change  and  deletion,  and 
error  correction. 


4.  Ensure  that  the  ease  of  user  actions 
will  match  desired  ends;  make  frequent 
or  urgent  actions  easy  to  take,  but  make 
potentially  destructive  actions 
sufficiently  difficult  that  they  will  require 
extra  user  attention. 


5.  Ensure  that  the  computer  changes  data 
only  as  a  result  of  explicit  actions  by  a 
user,  and  does  not  initiate  changes 
automatically. 

6.  For  conditions  which  may  require 
special  user  attention  to  protect  against 
data  loss,  provide  an  explicit  alarm 
and/or  warning  message  to  prompt 
appropriate  user  action. 

7.  Allow  users  to  UNDO  an  immediately 
preceding  control  action  that  may  have 
caused  an  unintended  data  loss. 


8.  Design  the  LOG-ON  process  and 
procedures  for  user  identification  to  be 
as  simple  as  possible  and  consistent  with 
protecting  data  from  unauthorized  use. 

9.  Design  the  LOG-ON  process  to 

provide  prompts  for  all  user  entries, 
including  passwords  and/or  whatever 
other  data  are  required  to  confirm  user 
identity  and  to  authorize  appropriate  data 
access/change  privileges. _ 
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_ Human  Factors  Principle _ 

10.  When  passwords  are  required,  allow 
users  to  choose  their  own  passwords. 

11.  Allow  users  to  change  their 
passwords  whenever  they  choose. 

12.  When  a  password  must  be  entered  by 
a  user,  ensure  that  password  entry  can  be 
private;  password  entries  should  not  be 
displayed. 

13.  Impose  a  maximum  limit  on  the 
number  and  rate  of  unsuccessful  LOG¬ 
ON  attempts  that  will  provide  a  margin 
for  user  error  while  protecting  the  system 
from  persistent  attempts  at  illegitimate 
access. 


14.  Once  a  user’s  identity  has  been 
authenticated,  ensure  that  whatever  data 
acccss/change  privileges  are  authorized 
for  that  user  will  continue  throughout  a 
work  session. 


15.  Establish  user  authorization  for  data 
access  at  initial  LOG-ON;  do  not  require 
further  authentication  when  a  user 
requests  display  of  particular  data. 

16.  When  displayed  data  are  classified 
for  security  purposes,  include  a 
prominent  indication  of  security 
classification  in  each  display. 

17.  When  confidential  information  is 
displayed  at  a  work  station  that  might  be 
viewed  by  casual  onlookers,  provide  the 
user  with  some  rapid  means  of 
temporarily  suppressing  a  current 
display,  and  then  resuming  work  later. 


Human  Factors  Principle 


Evaluation 


Acceptable  with 
Deficiencies 


Comments 


1.0  Viewing  distance.  A  400  mm  (16- 
inch)  viewing  distance  shall  be  provided 
whenever  practicable.  When  periods  of 
observation  will  be  short,  or  when  dim 
signals  must  be  detected,  the  viewing 
distance  may  be  reduced  to  250  mm  (10 
inches).  Design  should  permit  the 
observer  to  view  the  CRT  as  close  as 
desired. 


2.0  Reflected  glare.  Reflected  glare 
shall  be  minimized  by  proper  placement 
of  the  CRT  relative  to  the  light  source 
and  the  use  of  a  hood,  shield,  or  optical 
coating  on  the  CRT. 


3,0  Ambient  surfaces.  Surfaces  adjacent 
to  the  CRT  shall  have  a  dull  matte  finish. 


4.0  Font  legibility.  Where  alphanumeric 
characters  appear  on  CRT-like  displays, 
the  font  style  shall  allow  discrimination 
of  similar  characters,  such  as  the  letter  1 
and  the  number  1  and  the  letter  z  and  the 
number  2. 


Audio  Displays 


1.0  Type  of  voice.  The  voice  used  in 
recording  verbal  signals  shall  be 
distinctive  and  mature. 


2.0  Delivery  style.  Verbal  signal  shall 
be  presented  in  a  formal,  impersonal 


3.0  Message  content.  In  selecting  words 
to  be  used  in  audio  warning  signals, 
priority  shall  be  given  to  intelligibility, 
aptness,  and  conciseness  in  that  order. 
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Human  Factors  Principle 


Evaluation 


Acceptable  with 
Deficiencies 


Comments 


Equipment  Labeling 


1.0  Location.  The  gross  identifying 
label  on  a  unit,  assembly,  or  major 
subassembly  shall  be  located  externally, 
in  such  a  position  that  it  is  not  obscured 
by  adjacent  items;  on  the  flattest,  most 
uncluttered  surface  available;  and  on  a 
main  chassis  of  the  equipment. 


2.0  Functional  labeling.  Each  control 
and  display  shall  be  labeled  according  to 
function,  and  the  following  criteria  shall 
apply:  (a)  similar  names  for  different 
controls  and  displays  shall  be  avoided; 

(b)  instruments  shall  be  labeled  in  terms 
of  what  is  being  measured  or  controlled, 
taking  into  account  the  user  and  purpose; 

(c)  control  labeling  shall  indicate  the 
functional  result  of  control  movement 
(e.g.,  on,  off);  (d)  when  controls  and 
displays  must  be  used  together  (in  certain 
adjustment  tasks),  appropriate  labels 
shall  indicate  their  functional 
relationship  -  the  selection  and  use  of 
terminology  shall  be  consistent. 


3.0  Location,  (a)  Ease  of  control 
operation  shall  be  given  priority  over 
visibility  of  labels;  (b)  labels  would 
normally  be  placed  above  the  controls 
and  displays  they  describe.  When  the 
panel  is  above  eye  level,  labels  may  be 
located  below  if  label  visibility  will  be 
enhanced. 


Workspace  Design 


1.0  Kick  space.  All  cabinets,  consoles, 
and  work  surfaces  that  require  an 
operator  to  stand  or  sit  close  to  their 
front  surfaces  shall  contain  a  kick  space 
at  the  base  at  least  100  mm  (4  inches) 
deep  and  100  mm  (4  inches)  hich. 
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Appendix  C 
to  SPEARS  OT&ETEP 


SPEARS  OT&E  USABILITY  STRUCTURED  INTERVIEW 
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SPEARS  OT&E  USABILITY  STRUCTURED  INTERVIEW 


1.  Were  you  made  familiar  enough  with  the  SPEARS  device? 

2.  Did  you  receive  proper  training  on  the  SPEARS? 

3.  Do  you  think  that  the  SPEARS  will  help  you  perform  your  job? 

If  no,  explain  why. 

If  yes,  explain  why. 

4.  How  did  the  SPEARS  device  affect  your  ability  to  find  explosives  in  bags  ? 

5.  How  useful  was  the  SPEARS  feedback? 

6.  How  useful  was  the  SPEARS  Threat-Image  Projection  (TIP)  training  and  testing? 

7.  What  training  needs  improvement? 

8.  What  training  was  most  effective? 

9.  How  understandable  was  the  training? 

10.  How  does  the  SPEARS  effect  your  normal  screening  job? 

11.  How  does  the  SPEARS  effect  the  flow  of  baggage  and  passengers  through  the 
checkpoint? 

12.  Describe  how  difficult  or  easy  it  was  to  use  the  SPEARS  machine. 
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Appendix  D 
to  SPEARS  OT&ETEP 


SPEARS  IMAGE  CONTENT  CHECBCLIST 


D-1 


SPEARS  IMAGE  CONTENT  CHECKLIST 


Evaluation 

SPEARS  Image  Content 

Yes 

No 

1.  The  number  of  images  of 
CTIs  provided  by  the  system  is 

NA 

NA 

2.  The  number  of  images  of 
FTIs  provided  by  the  system  is 

NA 

NA 

3.  The  number  of  CTIs  of 
weapons  is 

NA 

NA 

4.  The  number  of  CTIs  of 
explosives  is 

NA 

NA 

5.  The  number  of  CTIs  of 
incendiary  devices  is 

NA 

NA 

6.  The  number  of  CTIs  of 
dangerous/deadly  devices  other 
than  weapons,  incendiaries,  or 
explosives  is 

NA 

NA 

7.  The  number  of  FTIs  of 
weapons  is 

NA 

NA 

8.  The  number  of  FTIs  of 
explosives  is 

NA 

NA 

9.  The  number  of  FTIs  of 
dangerous/deadly  devices  other 
than  weapons,  incendiaries,  or 
explosives  is 

NA 

NA 

10.  At  least  two  different 
aspect  angles  are  available  for 
each  Cn  presentation. 

11.  At  least  two  different 
aspect  angles  are  available  for 
each  FTI  presentation. 
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to  SPEARS  OT&ETEP 


SPEARS  CUSTOMIZATION  CHECKLIST 


E-1 


SPEARS  CUSTOMIZATION  CHECKLIST 


Evaluation 

SPEARS  Customization 

Yes  No 

Comments/Description 

1.  Combined  Threats  and  Images 
(CTIs)  can  be  selected  for  each 
screener  as  a  function  of  threat 
category. 

2.  CTIs  can  be  selected  for  each 
screener  as  a  function  of  difficulty. 

3.  CTIs  can  be  selected  for  each 
screener  as  a  function  of  aspect 
angle. 

4.  CTIs  can  be  selected  as  a 
function  of  the  combination  of 
threat  category,  screener  difficulty, 
and  aspect  angle. 

5.  CTIs  can  be  archived  with  the 
associated  screener  outcome  ( hit, 
miss,  false  alarm,  and  correct 
rejection). 

6.  Each  archived  CTI  with 
associated  screener  trial  outcome 
includes  date  and  time  of  day. 

7.  Fictional  Threat  Images  (PT'ls) 
can  be  selected  for  each  screener  as 
a  function  of  threat  category. 

8.  FTIs  can  be  selected  for  each 
screener  as  a  function  of  difficulty. 

9.  FTIs  can  be  selected  for  each 
screener  as  a  function  of  aspect 
angle. 
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Appendix  F 
to  SPEARS  OT&E  TEP 


SPEARS  FEEDBACK  CHECKLIST 
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SPEARS  FEEDBACK  CHECKLIST 


Evaluation 

SPEARS  Feedback 

Yes 

No 

1.  Feedback  is  immediate  after 
correct,  incorrect,  or  missed 
identification  of  an  FTI. 

2.  Feedback  is  displayed  in  a 
consistent  position. 

3.  Feedback  is  consistent  in  terms 
of  format  and  content. 

4.  Feedback  is  provided  for  correct 
FTI  identification. 

5.  Feedback  is  provided  for  missed 
FTI  identification. 

6.  Following  missed  FTI 
identification,  the  location  of  the 
missed  FTI  is  provided  on  the 
display. 

7.  Feedback  is  provided  for  threat 
items  that  do  not  contain  threat 
images. 

8.  FTIs  are  removed  from  the 
display  to  allow  additional  image 
scanning. 

9.  The  time  interval  between  user 

NA 

i 

NA 

initiation  and  system  feedback  for 
each  condition  of  correct  or  missed 
identification,  image  removal,  and 
threat  items  which  do  not  contain 
threat  images  is 

SPEARS  Feedback 


10.  The  nature  of  the  feedback  (i.e., 
visual  and/or  auditory,  graphical, 
textual,  etc.)  for  each  condition  of 
correct  or  missed  identification, 
image  removal,  and  threat  items 
which  do  not  contain  threat  images 
is 


Evaluation 

Yes 

No 

NA 

NA 

Appendix  G 
to  SPEARS  OT&ETEP 


SPEARS  CAPABILITIES  SUMMARIES  CHECKLIST 
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SPEARS  CAPABILITIES  SUMMARIES  CHECKLIST 


SPEARS  Capabilities 


1.  Summary  reports  are 
automatically  generated. 

2.  Summary  reports  are  generated 
on  demand. 

3.  Summary  reports  contain 
descriptive  statistics. 

4.  Descriptive  statistics  are 
categorized  according  to: 

a.  threats  presented  and 
outcomes 

b.  screener  identification 

c.  checkpoint  activity  level 

d.  time  of  day 

5.  Summary  reports  are  readily 
understandable  by  the  user  (based 
on  structured  interview  question 
results). 

6.  Performance  levels  for 
individual  screeners  per  screening 
session  are  actively  and  accurately 
reported  as  a  function  of  the 
probability  of  detection  (Pd). 

7.  Performance  level  reports  can  be 
provided  to  a  remotely  situated 
supervisor. 

8.  A  remotely  situated  supervisor 
can  be  notified  when  performance 
levels  are  above  or  below  supervisor 
preset  performance  criteria 
thresholds. 
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INTEROPERABILITY  CHECKUST 
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INTEROPERABILITY  CHECKLIST 


Evaluation 

SPEARS  Interoperability 

Yes 

No 

1.  A  connection  via  modem  can  be 
established  between  two  Screener 
Proficiency  Evaluation  and 

Reporting  System  (SPEARS) 
devices  or  a  SPEARS  device  and  a 
computer  over  standard  telephone 
lines. 

2.  The  SPEARS  remote  connection 
requires  no  more  than  two  people 
(one  on-site,  one  at  the  remote  site) 
to  operate. 

3.  New  CTIs  can  be  downloaded 
remotely  to  the  SPEARS  on-site 
device. 

4.  New  FTIs  can  be  downloaded 
to  the  SPEARS  on-site  device. 

5.  Status  reports  can  be  initiated 
remotely  via  the  modem  connection. 

6.  Status  report  data  can  be 
accessed  remotely  and  saved  to  a 
local  computer  Ele. 

7.  Status  reports  can  be  printed  out 
from  the  remote  location. 

8.  Remotely  accessed  status  reports 
are  similar  to  those  available  on-site 
from  the  SPEARS  device. 

9.  The  time  required  to  transfer  a 

NA 

NA 

Cn  remotely  is 

10.  The  time  required  to  transfer  an 
Fn  remotely  is _ . 


NA 


NA 


Appendix  I 
to  SPEARS  OT&ETEP 


SECURITY  ACCESS  CONTROL  CHECKLIST 


I-l 


SECURITY  ACCESS  CONTROL  CHECKLIST 
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12.  Users  are  required  to  enter  identification  before  performing  any 
actions. 


Evaluation 

SPEARS  Security  Access  Control _  Pass  Fail  Comments/Description 
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23.  The  manual  provides  guidelines  on  the  consistent  and  effective 
use  of  the  protection  features  of  the  system,  how  they  interact,  and 
how  to  successfully  generate  a  new  secure  system. 


Appendix  J 
to  SPEARS  OT«&ETEP 


SPEARS  IMAGE  INSERTION  ISSUE  CHECKLIST 
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SPEARS  IMAGE  INSERTION  ISSUE  CHECKLIST 


SPEARS  Insertion 


1.  FTIs  can  be  generated  using  a 
computer  generated  threat  object 
and  neutral  baggage  image. 

2.  FTIs  can  be  automatically 
generated. 

3.  FTIs  are  not  predictable  in  the 
order  they  are  presented. 

4.  CTIs  can  be  inserted  into  the 
normal  baggage  flow. 

5.  CTIs  can  be  automatically 
generated. 

6.  CTIs  are  not  predictable  in  the 
order  they  are  presented. 

7.  When  an  FTl  and  neutral  bag 
image  are  combined,  FTIs 
perceptually  appear  inside  the 
neutral  bag. 

8.  Fn  position  in  the  neutral  bag 
can  be  randomly  determined. 

9.  Fn  orientation  in  the  neutral 
bag  can  be  controlled. 


Evaluation 


Yes  No  I  Comments/Description 


Appendix  K 
to  SPEARS  OT&E  TEP 


EMBRY-RIDDLE  AERONAUTICAL  UNIVERSITY 
AIRPORT  SECURITY  SCREENING  PROGRAM  PROTOCOL 
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HIDDEN  HGURES  TEST 


This  version  of  the  Hidden  Figures  Test  (HFT)  is  a  two-part  instrument  with  32  items  equally 
distributed  between  both  parts.  The  instrument  is  a  computer-based  administration  of  a 
published  paper-and-pencil  test.  Computer-based  administration  was  selected  to  permit  the 
assessment  of  reaction  time  measures,  in  addition  to  the  standard  accuracy  measure. 

The  instrument  is  introduced  to  subjects  as  “a  test  of  your  ability  to  find  which  one  of  five  simple 
figures  can  be  found  in  a  more  complex  pattern.”  The  administration  is  completely  automated 
and  requires  no  experimenter  intervention  once  initialized.  Unlike  the  paper-and-pencil 
instrument,  only  one  test  item  is  presented  at  a  time.  Subsequent  test  items  are  not  presented 
until  a  response  has  been  entered  for  the  previous  test  item,  at  which  time  the  item  responded  to 
is  permanently  removed  from  the  computer  monitor.  Each  item  presentation  maintains  the 
response  set  of  five  target  patterns  horizontally  across  the  top  of  the  computer  monitor.  Each  of 
the  five  response  targets  are  numerically  labeled  from  1  through  5  directly  beneath  the 
corresponding  target  patterns.  The  response  set  remains  on  the  monitor  any  time  the  actual  test 
administration  is  in  progress  and  a  test  item  is  present.  The  complex  background  patterns  are 
presented  beneath  the  response  set  at  approximately  the  center  of  the  screen.  Identical  to  the 
paper-and-pencil  HFT,  the  complex  background  patterns  vary  in  size,  shape,  and  complexity. 

The  current  version  also  differs  from  the  original  version  in  that  the  examples  are  dynamically 
displayed.  Examples  are  displayed  for  subjects  by  first  presenting  the  complex  background 
pattern  and  then  highlighting  the  target  figure  by  use  of  color  traced  over  the  background  pattern. 
Subjects  are  directed  to  respond  by  entering  their  choice  on  the  computer  numeric  keypad. 

This  version  of  the  instrument  is  further  distinguished  by  having  the  subjects  perform  two 
examples.  After  an  example  is  responded  to,  feedback  is  provided.  For  both  correct  and 
incorrect  responses,  the  correct  response  (target  pattern)  is  highlighted  in  color  on  the 
background  pattern.  Positive  responses  are  provided  with  the  textual  feedback,  ‘That’s  correct” 
on  the  monitor.  Incorrect  responses  are  provided  feedback  on  the  monitor  with  the  text  ‘‘Sorry, 
only  the  correct  response  number  figure  can  be  found  in  the  pattern.” 

Several  other  directions  appear  interspersed  in  the  instruction  set  and  are  provided  to  the  subjects 
before  they  begin  the  actual  test.  These  are  all  provided  textually  via  the  computer  software  and 
are  subject-paced: 

a.  Subjects  are  informed  that  there  is  only  one  of  the  target  figures  in  each  complex 

pattern. 

b.  Each  target  figure  will  always  be  right  side  up  and  exactly  the  same  size  as  the  one 
in  the  complex  pattern. 

c.  No  additional  lines  can  be  added  to  the  complex  pattern.  The  target  figure  must 
be  traceable  on  the  existing  complex  pattern.  (This  is  further  demonstrated  graphically  in 
conjunction  with  the  textual  instruction.) 
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d.  A  score  is  calculated  by  subtracting  the  number  of  incorrect  responses  from  the 
number  of  correct  responses.  Subjects  are  informed  that  it  is  not  to  their  advantage  to  guess, 
unless  they  are  able  to  eliminate  at  least  some  of  the  possible  responses  that  they  know  are 
wrong. 


e.  Information  is  provided  that  advises  there  is  no  time  limit,  but  to  work  quickly 
since  response  times  are  assessed  and  are,  therefore,  important. 

f.  The  number  of  test  items  is  clearly  identified. 

Since  reaction  times  are  recorded,  each  part  of  the  test  begins  with  a  10-second  countdown  that  is 
prominently  displayed  in  the  center  of  the  monitor  with  the  preface  “Test  will  begin  in  _ 
seconds.”  The  textual  command  “Begin”  appears  1  second  after  the  countdown  is  completed, 
followed  500  milliseconds  later  by  the  first  test  item.  A  10-second  rest  period  occurs  between 
the  first  and  second  parts  of  the  test. 


RESULTS 


Introduction 


Several  analyses  were  conducted  on  both  instruments  to  assess  the  psychometric  properties  of 
each  one.  These  analyses  were  important  to  determine  if  the  transition  from  a  paper-and-pencil 
format  to  a  computer-based  presentation  adversely  impacted  test  characteristics  of  key  interest: 

a.  The  capability  to  identify  individual  differences. 

b.  The  presence  of  speed-accuracy  trade-offs. 

c.  The  internal  consistency  of  the  instruments. 

Post-test  subject  debriefing  data  were  also  examined  to  determine  if  alternate  strategies  were 
employed  by  the  subjects  to  circumvent  the  assessment  of  the  intended  abilities. 

Hidden  Figures  Test 


This  instrument  is  divided  into  two  parts,  each  with  16  items.  The  computer-based  version 
maintained  this  original  structure.  The  mean  and  standard  deviations  for  the  accuracy  score  were 
M=7.00  (SD=9.03)  and  M=5.13  (SD=8.70)  for  parts  one  and  two,  respectively.  These  data 
indicate  that  the  instrument  is  sensitive  to  individual  differences  and  demonstrates  a  mean  to 
standard  deviation  ratio  greater  than  1:1.25.  Correct  and  incorrect  reaction  times  were  M=47.69 
seconds  (SD-  23.43)  and  M=72.84  seconds  (SD=48.47),  respectively. 

An  examination  of  individual  differences  was  further  pursued  by  comparing  mean  accuracy 
scores  of  both  parts  of  the  instrument  by  dividing  the  sample  into  upper  and  lower  quartile 
groups  using  the  accuracy  score  data.  The  mean  accuracy  scores  for  both  parts  of  the  instrument 
for  the  upper  quartile  group  were  M=16.00  and  M=13.14.  Respective  mean  accuracy  scores  for 
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the  lower  quartile  group  were  M=6.29  and  M=-8.86.  A  t-test  for  differences  yielded 
t(l,12)=18.15,  p>.001  for  part  one,  and  t(l,12)=20.14,  p>.001  for  part  two  of  the  instrument  in 
comparing  quartile  groups.  The  computer-based  version  of  the  HFT  is  effective  in 
discriminating  performance  among  examinees. 

An  item  analysis  was  conducted  between  the  quartile  groups  to  determine  if  any  test  items  were 
not  useful  in  discriminating  performance.  This  analysis  indicated  that  all  32  items  were 
predictive  of  performance  differences,  albeit  of  different  magnitudes.  All  but  two  items  resulted 
in  differences  at  a  minimum  of  the  50  percent  level.  That  is  to  say,  at  least  half  of  the  lower 
quartile  group  scored  incorrectly  on  all  items  that  the  upper  quartile  group  responded  to  correctly. 
Based  on  these  results,  all  32  items  were  retained  for  the  finalized  instrument. 

A  split-half  reliability  analysis  was  also  conducted  to  assess  the  internal  consistency  of  the 
instrument.  A  Pearson  Product  Moment  correlation  coefficient  between  the  first  and  second 
parts  of  the  instrument  demonstrated  =.92(p>.01). 

The  presence  of  a  speed-accuracy  trade-off  strategy  in  performing  the  test  was  assessed  by 
examining  the  relationship  between  the  accuracy  score  for  both  parts  of  the  test  combined  and 
mean  correct  reaction  time  across  all  32  items.  A  Pearson  correlation  coefficient  of  r=.24 
indicates  only  a  weak  relationship  between  these  two  performance  measures.  There  appears  to  be 
little  reason  to  suspect  that  subjects  sacrificed  speed  for  accuracy  or  vice-versa.  This  was 
anticipated,  given  that  the  test  instructions  directed  subjects  to  maintain  a  balance  between  these 
two  performance  goals. 
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HIDDEN  PATTERNS  TEST 


The  Hidden  Pattern  Test  (HPT)  was  developed  as  a  two-part  instrument,  with  100  items  equally 
distributed  between  both  parts.  Test  items  do  not  increase  in  difficulty  level  throughout  the  test. 
This  assessment  tool  is  a  computer-based  administration  of  that  developed  by  the  Educational 
Testing  Service.  This  revision  of  the  instrument  permits  assessment  of  both  accuracy  and 
response  time  measures. 

The  HPT  is  introduced  as  a  test  of  the  ability  to  recognize  a  figure  that  is  hidden  among  other 
lines.  Instructions  inform  subjects  that  speed  of  recognition  and  response  is  important,  but  not  to 
sacrifice  accuracy. 

Similar  to  the  HFT,  the  instrument  is  fully  automated  and  self-contained.  Test  items  are 
displayed  one  at  a  time  on  the  computer  monitor  and  are  removed  once  a  corresponding  response 
is  made.  Each  trial  permanently  displays  the  target  figure  prominently  in  the  center  of  the 
monitor  any  time  the  test  administration  is  in  progress.  The  target  figure  (to  which  the  test  item 
is  compared)  appears  3  centimeters  below  the  test  item.  Both  test  items  and  target  figures  are 
identical  to  those  used  in  the  original  paper-and-pencil  instrument. 

The  design  of  the  instrument  is  similar  to  the  automated  HFT  in  several  ways:  examples  are 
dynamically  presented;  examples  use  a  color  trace  of  the  test  item  over  the  target  figure  for 
illustration;  a  red  “x”  is  displayed  during  example  items  when  the  test  item  cannot  be  found  in 
the  target  figure;  and  all  responses  are  made  using  a  numeric  keypad.  Seven  examples  of  the  test 
item  appearing  within  the  target  figure  are  provided,  as  are  four  examples  when  a  match  is  not 
present.  All  examples  are  self-paced  and  are  presented  in  multiple  orientations. 

The  instructions  further  inform  subjects  that  the  test  item  may  be  found  in  the  target  figure  in  an 
upside  down,  right-side  up,  or  rotated  configuration.  Similar  to  the  HFT,  an  individual’s 
performance  is  evaluated  as  the  number  correct  minus  the  number  wrong.  Subjects  are  advised 
that  it  is  not  to  their  advantage  to  guess. 

Each  part  of  the  test  begins  with  a  10-second  countdown  identical  to  that  used  with  the  HFT.  A 
brief,  subject-controlled,  rest  period  also  occurs  between  the  two  parts  of  the  test. 

Only  response  keys  “1”  and  “2”  are  activated  after  the  instruction  set  terminates.  Subjects  are 
instructed  to  depress  the  number  “1”  key  for  those  patterns  in  which  the  model  appears,  and  the 
number  “2”  key  for  those  patterns  in  which  the  model  does  not  appear. 
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Appendix  L 
to  SPEARS  OT&E  TEP 


SUBJECT  INFORMED  CONSENT  FORM 


INFORMED  CONSENT 


I, _ ,  have  received  a  briefing  by  the  FAA 

representative  as  to  the  purpose  of  the  FAA  study,  I  fully  understand  the  purpose  of  the  study 
and  have  been  provided  with  the  opportunity  to  ask  questions  of  the  FAA  representative.  The 
FAA  representative  informed  me  that  the  study  will  require  approximately  3  hours  of  training, 
followed  by  3  baggage  screening  trials,  and  30  minutes  to  complete  a  vision  test  and  1  hour  to 
complete  a  hidden  figures  test. 

I  understand  that  this  study  will  impose  very  little  stress.  The  only  stress  I  may  experience  in  this 
experiment  may  be  some  initial  frustration  as  I  leam  how  to  use  the  training  system.  As  part  of 
the  data  analysis,  my  data  will  be  combined  with  that  of  other  individuals  and  I  will  no  longer  be 
identifiable  as  a  participant.  I  have  been  informed  that  my  name  will  remain  CONFIDENTIAL, 

I  have  been  informed  that  I  have  the  right  to  withdraw  from  the  experiment,  and  that  the 
experiment  monitor  may  terminate  my  participation  in  the  interest  of  safety  and  the  experiment. 
I  also  certify  that  I  am  at  least  18  years  of  age. 

I  have  been  informed  that  if  additional  details  are  needed,  I  may  contact  any  of  the  test 
administrators  at  the  airport  during  the  study,  or  contact  James  L.  Fobes,  Ph.D.,  (609)  485-4944; 
or  Robert  L.  Malone,  (609)  645-0900,  upon  completion  of  the  study. 


Signed:  _ 

Date:  _ / _ / 


Witness:  _ 

Date:  _ / _ / 
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Appendix  M 
to  SPEARS  OT&ETEP 


SCREENER  PERSONAL  INFORMATION  FORM 
(SPEARS  OT&E  SCREENER  QUESTIONNAIRE) 
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FAA 


SCREENER  PROFICIENCY  EVALUATION  AND  REPORTING  SYSTEM 
OPERATIONAL  TEST  AND  EVALUATION 
SCREENER  QUESTIONNAIRE 

DATE: _  SUBJECT  NUMBER: _ ^ _ 

1.  What  is  your  sex? 

Male _  Female _ 

2.  How  long  have  you  been  a  baggage  screener? 

_ Years  _ Months 

3.  How  long  have  you  been  using  X-ray  equipment  to  screen  baggage? 

_ ^Years  _ Months 

4.  What  is  the  highest  level  of  education  or  degree  that  you  have  completed. 

8th  Grade  or  Less  _ 

Some  High  School  _ 

High  School  Graduate  _ 

Some  College  or  University  _ 

College  or  University  Graduate  _ 

5.  Do  you  wear  glasses  while  using  the  X-ray  equipment? 

Yes  No  _ 
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FAA 


SCREENER  PROFICIENCY  EVALUATION  AND  REPORTING  SYSTEM 
OPERATIONAL  TEST  AND  EVALUATION 
SCREENER  QUESTIONNAIRE 
(Continued) 

6.  Do  you  consider  English  to  be  your  primary  language? 

Yes _  No _ 

If  not,  in  which  language  are  you  most  proficient?  _ 

If  not,  are  you  proficient  with  English?  _ 
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Appendix  N 
to  SPEARS  OT&ETEP 


PROTOCOL  IMPROVISED  EXPLOSIVE  DEVICE  (lED)  DETECTION  TEST 


PROTOCOL 
lED  DETECTION  TEST 


READ  TO  SCREENERS: 

This  is  a  test  of  how  well  X-ray  machines  work  for  detecting  Improvised  Explosive  Devices 
(lEDs).  For  this  activity,  we  have  put  X-ray  images  of  passenger  bags  in  this  computer.  You 
will  view  the  X-ray  images,  one  bag  at  a  time,  and  inspect  each  bag  for  an  lED.  The  images  will 
be  displayed  on  the  video  monitor. 

INSTRUCTIONS  TO  SUBJECTS 

After  you  have  determined  that  your  subject  number  on  the  monitor  is  correct,  press  the 
“ENTER”  key  on  the  keyboard  to  start  the  practice  test.  Initially,  you  will  be  given  10  images  to 
practice  on  before  beginning  the  real  test.  Once  you  see  the  first  X-ray  image  appear  on  the 
monitor  screen,  your  task  will  be  twofold; 

First,  you  will  respond  to  the  question:  “Is  there  an  lED  in  this  X-ray  image?”  (The  question 
will  be  shown  at  the  bottom  center  of  the  monitor  frame.) 

To  indicate  “yes,”  press  the  key  labeled  “yes”  on  the  keyboard. 

To  indicate  “no,”  press  the  key  labeled  “no”  on  the  keyboard. 

Should  you  press  the  wrong  key,  an  error  message  will  appear  on  the  monitor  telling  you  that  you 
pressed  the  wrong  key. 

Because  we  want  to  know  how  well  the  machines  work,  it  is  important  that  you  answer  each 
question  to  the  best  of  your  ability.  It  is  just  as  important  for  you  to  say  “no”  when  you  do  not 
see  an  lED  as  it  is  to  say  “yes”  when  you  do  see  an  lED. 

You  will  have  10  seconds  to  make  your  response.  However,  you  should  answer  as  quickly  as 
possible.  Once  you  have  made  your  response,  you  will  not  allowed  to  go  back  and  change  it. 

If  you  have  not  answered  the  question  after  6  seconds,  an  audible  alarm  will  sound,  alerting  you 
that  you  have  only  4  seconds  left  to  answer  the  question. 

If  you  still  have  not  responded  after  10  seconds,  the  image  will  disappear  from  the  monitor 
screen  and  will  be  replaced  by  a  prompt  which  advises  you  that,  “Time  has  expired,  please 
answer.” 
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After  you  have  responded  to  the  first  question,  a  second  question  will  be  displayed  on  the  video 
monitor: 


“How  sure  are  you?” 
yes  =  more  sure  no  =  less  sure 

Respond  by  pressing  the  “yes”  or  “no”  key  on  the  keyboard  or  keypad,  regardless  of  whether 
your  answer  to  the  first  question  was  “yes”  or  “no”.  In  either  case,  you  will  use  the  same  two 
keys  that  you  used  to  respond  to  the  first  question. 

You  will  have  5  seconds  to  make  this  second  response.  If  you  have  not  yet  answered  this  second 
question  after  5  seconds,  the  question  will  disappear  from  the  monitor  screen  and  the  prompt 
appear  again  telling  you:  “Time  has  expired,  please  answer.”  The  system  will  not  forward  to  the 
next  image  until  you  have  made  your  response. 

You  will  repeat  this  procedure  for  each  image  until  all  images  have  been  viewed.  A  message  will 
then  appear  on  the  monitor  indicating  that  the  test  has  ended.  The  test  should  take  a  little  over  an 
hour  to  complete. 

Do  you  have  any  questions? 

Thank  you  for  participating  in  this  test.  Press  the  “ENTER”  key  on  your  keyboard  to  start  the 
practice  test.  When  you  have  completed  the  practice  test,  press  the  “ENTER”  key  again  to  start 
the  real  test. 
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Appendix  O 
to  SPEARS  OT&E  TEP 


SPEARS  ON-LINE  OT&E 

STATEMENT  OF  DUTIES  FOR  ADMINISTRATIVE  PERSONNEL 
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SPEARS  ON-LINE  OT«S:E 

STATEMENT  OF  DUTIES  FOR  ADMINISTRATIVE  PERSONNEL 


Test  Director: 

Responsible  for  directing  and  overseeing  all  test  activities  and  personnel. 

Federal  Aviation  Administration 


Test  Manager: 

Ensures  that  screeners  are  greeted  and  completes  required  administration. 
Conducts  briefs  and  debriefs. 

Manages  daily  test  activities  and  starts  and  stops  the  test  sequence  as  required. 
Plans  for  and  directs  contingency  activities. 

Liaisons  with  security  company  administrative  personnel. 

Liaisons  with  airline  personnel. 

Ensures  that  the  proper  screener  is  in  the  required  location  at  the  required  time. 
Resolves  any  encountered  problems  with  the  test  director. 

Executes  all  required  logistical  activities. 


Galaxy  Scientific  Corporation 


Pretest  Data  Collector 


Terminal  2  data  collectors  train  screeners  on  procedures  and  use  of  TnT^“  device. 

Conducts  contrast  acuity  tests  on  screeners  using  the  required  test  forms. 

Ensures  that  the  screener’s  number  is  recorded  on  test  forms. 

Ensures  that  test  forms  are  given  to  Galaxy  Scientific  Corporation  personnel. 

Ensures  that  each  screener  completes  a  questionnaire. 

Briefs  and  debriefs  screeners  regarding  Improvised  Explosive  Device  (lED)  detection  test. 
Provides  guidance  and  assistance  to  screeners  on  matters  pertaining  to  conduct  of  test. 
Administers  lED  detection  test. 

Escorts  screeners  between  security  checkpoint  and  training  and  testing  rooms. 

Briefs  and  debriefs  screeners  regarding  Hidden  Figures/Patterns  test. 

Provides  guidance  and  assistance  to  screeners  on  matters  pertaining  to  conduct  of  test. 
Administers  Hidden  Figures/Patterns  test. 


Federal  Aviation  Administration 
Galaxy  Scientific  Corporation 
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Posttest  1  Data  Collector 


Briefs  and  debriefs  screeners  regarding  lED  detection  test. 

Provides  guidance  and  assistance  to  screeners  on  matters  pertaining  to  conduct  of  test. 
Administers  lED  detection  test. 

Escorts  screeners  between  security  checkpoint  and  training  and  testing  rooms. 

Federal  Aviation  Administration 
_  Galaxy  Scientific  Corporation 


Posttest  2  Data  Collector  -  lED 


Briefs  and  debriefs  screeners  regarding  lED  detection  test. 

Provides  guidance  and  assistance  to  screeners  on  matters  pertaining  to  conduct  of  test. 
Administers  lED  detection  test. 

Escorts  screeners  between  security  checkpoint  and  training  and  testing  rooms. 

Federal  Aviation  Administration 
Galaxy  Scientific  Corporation 


Posttest  2  Data  Collector  -  TnT'*'” 


Terminal  2  data  collector  briefs  and  debriefs  screeners  regarding  usability  assessment. 
Provides  guidance  and  assistance  to  screeners  in  completing  usability  interview. 


Terminal  7  data  collector  briefs  and  debriefs  screeners  regarding  insertion  representativeness 
assessment. 

Provides  guidance  and  assistance  to  screeners  in  completing  insertion  representativeness  ratings. 

Federal  Aviation  Administration 
Galaxy  Scientific  Corporation 
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SPEARS  ON-LINE  OT&E  MASTER  SCHEDULE 
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SPEARS  ON-LINE  OT&E  MASTER  SCHEDULE 


Appendix  Q 
to  SPEARS  OT&ETEP 


SPEARS  DETAILED  TEST  SCHEDULES 
(PRETEST,  POSTTEST 1,  AND  POSTTEST  2) 
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SPEARS  ON-LINE  OT&E  PRE-TEST 
TRAINING  GROUP 


Q-2 


SPEARS  ON-LINE  OT&E  PRETEST 
CONTROL  GROUP 


•  •  Documentation  A - ►  Hidden  Figures 


lED  Testing 
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SPEARS  ON-LINE  OT&E  POSTTEST  1 
TRAINING  GROUP 


SPEARS  ON-LINE  OT&E  POSTTEST  1 
CONTROL  GROUP 


SPEARS  ON-LINE  OT&E  POSTTEST  2 


Note:  Training  Group  Day  1-2,  Control  Group  Day  3-5 


Appendix  R 
to  SPEARS  OT&ETEP 


SPEARS  ON-LINE  OPERATIONAL  TEST  AND  EVALUATION  (OT«&E) 
SCREENER  PERSONNEL  SCHEDULE 
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SPEARS  ON-LINE  OPERATIONAL  TEST  AND  EVALUATION  (OT&E) 
ABBREVIATED  SCREENER  PERSONNEL  SCHEDULE 
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NOTE:  This  test  schedule  has  been  abbreviated  due  to  space  constaints.  However,  a  complete  copy  of  the  test  schedule  will  be 
provided  upon  request. 


All  Subjects  Day  1  1530  to  1630  Aloha  Room  In  Brief 

1630  to  1700  Aloha  Room  Documentation 
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REGAN  HIGH  CONTRAST  ACUITY  CHARTS 


SCORE  SHEET 
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It  is  important  to  urge  the  patient  to  guess  each  letter,  even  when  uncertain. 
Mark  each  error  by  crossing  out  each  letter  missed. 


Appendix  U 
to  SPEARS  OT&ETEP 


SCREENER  INSTRUCTIONS 


U-1 


SCREENER  OPERATING  INSTRUCTIONS  FOR  TnT™  TIP 


These  instructions  are  intended  to  help  you  operate  the  Linescan  System  E-Scan  X-ray  machine 
when  undergoing  on-line  training  and  testing  during  this  study.  As  you  know,  the  X-ray  machine 
is  used  to  screen  hand-carried  parcels  to  identify  security  threats.  A  new  training  device  has  been 
developed  to  help  screeners  such  as  yourself  learn  to  use  the  E-Scan  X-ray  machine  by  taking 
self-paced  training  lessons  and  on-line  training  and  testing  offered  by  the  Linescan  Training  and 
Testing  system.  (Throughout  the  rest  of  this  guide,  the  term  Linescan  Tn’P^  is  used  to  refer  to 
the  training  and  testing  system.) 

U.l  WHAT  IS  LINESCAN  TnTTM? 

The  Linescan  TnT'^“  training  system  is  a  training  tool  developed  to  help  you  learn  how  to  operate 
the  E-Scan  X-ray  machine  and  how  to  identify  security  threats. 

The  training  system  itself  is  a  combination  of  equipment  and  software  (computer  programs). 

U.2  PURPOSE  OF  THESE  OPERATING  INSTRUCTIONS. 

The  purpose  of  these  operating  instructions  is  to  provide  you  with  all  the  information  you  need  to 
use  Linescan  TnT™  in  the  on-line  training  and  testing  mode.  With  it,  you  will  learn  how  to: 

a.  log  onto  the  system, 

b.  use  Linescan  TnT'''”  with  the  X-ray  machine  for  on-the-job  practice,  and 

c.  tog  out  of  the  system. 

These  instructions  are  not  intended  to  teach  you  how  to  operate  the  E-Scan  X-ray  system  or  how 
to  identify  security  threats. 

U.3  CONTENTS  OF  THESE  INSTRUCTIONS. 


These  instructions  cover  the  topics  listed  below: 

a.  Starting  up  the  equipment. 

b.  Logging  onto  Linescan  TnT'''”. 

c.  The  Main  Menu  and  the  basics  of  using  Linescan  TnT'''”. 

d.  Instructions  for  using  Linescan  TnT^”  when  it  is  connected  to  the  E-Scan  X-ray 
machine  for  on-line  training  and  testing  when  screening  for  parcels. 

e.  Logging  out  of  Linescan  TnT'''” . 
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U.4  WHAT  IS  ON-LINE  TRAINING  AND  TESTING? 


During  on-line  training  and  testing,  you  practice  your  skills  in  threat  identification  while 
operating  the  actual  X-ray  machine.  You  also  learn  what  to  do  when  you  find  a  threat.  On-line 
training  and  testing  is  conducted  during  an  actual  work  shift. 

For  on-line  training  and  testing,  Linescan  TnT’'”  is  connected  to  the  E-Scan  X-ray  machine.  Your 
goal  is  to  identify  all  threats  that  appear  to  pass  through  the  X-ray  machine.  Linescan  TnT'‘‘“ 
then  asks  you  to  locate  the  specific  threat  you  see  in  the  X-ray  image  and  to  classify  it  as  a  gun,  a 
knife,  an  explosive  device,  or  a  miscellaneous  weapon.  Then  Linescan  TnT'*'”  displays  a  security 
procedure  for  you  that  tells  you  how  to  handle  the  kind  of  threat  you  have  detected. 

In  on-line  testing,  your  performance  is  recorded  so  that  your  manager  can  check  your  progress  to 
see  where  your  strengths  and  weaknesses  are.  This  recording  feature  will  be  used  by  the  FAA 
representatives  to  see  how  well  the  Linescan  TnT'*'”  trains  screeners  to  find  threats.  The 
recordings  are  confidential,  and  no  one  will  know  your  results. 

U.5  GETTING  STARTED. 

This  section  tells  you  how  to  log  onto  the  Linescan  TnT'‘'“  training  system  as  an  authorized  user. 
It  also  introduces  the  functions  available  to  you  as  an  operator  and  covers  the  basics  of  using  the 
Linescan  TnT'*'”  screens.  Finally,  it  tells  you  how  to  log  out  at  the  end  of  your  work  session. 

U.5.1  Using  the  Trackball. 

The  most  important  part  of  learning  to  use  Linescan  TnT^“  is  learning  to  use  the  trackball.  Take 
a  few  moments  now  to  become  familiar  with  using  it. 

Pointing 

The  trackball  is  a  pointing  device.  As  you  roll  the  trackball,  a  pointer  (usually  but  not  always  an 
arrow)  moves  on  the  screen  in  correspondence  with  the  trackball’s  movement.  You  use  the 
trackball  to  point  at  objects  on  the  screen. 

Sometimes  the  pointer  is  not  an  arrow.  Sometimes  it  takes  the  shape  of  a  pointing  hand,  usually 
when  it  is  over  a  certain  type  of  object  on  the  screen.  Other  times,  the  pointer  takes  the  shape  of 
a  wrist  watch  with  hands  that  spin  clockwise.  This  “wrist  watch”  indicates  that  the  computer  is 
processing  your  request  and  you  should  wait  momentarily. 

Clicking 

In  order  to  tell  Linescan  TnT™  what  you  want  it  to  do,  you  have  to  do  more  than  simply  point  at 
an  object.  To  select  the  object  (that  is,  to  tell  the  computer  you  want  to  use  that  particular 
object),  you  click  on  it. 
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To  click  on  an  object  that  you  have  pointed  at,  press  the  pad  on  either  side  of  the  trackball.  You 
will  feel  a  slight  “clicking”  pressure,  almost  as  if  you  have  pressed  a  button,  and  you  will  hear  a 
“click.” 

You  use  the  actions  of  pointing  and  clicking  throughout  your  interaction  with  Linescan  TnT'''”. 
When  this  guide  instructs  you  to  select  or  to  “click”  or  “click  on”  an  object,  you  must  first  point 
at  the  object,  then  press  the  trackball  pad. 

U.5.2  Simulated  Control  Panel. 

You  will  often  see  on  a  Linescan  TnT’’”  screen  a  control  panel  that  looks  like  the  control  panel 
on  the  actual  TnT'*'”  machine.  To  use  the  simulated  control  panel  on  the  screen  (figure  U-1),  you 
will  use  the  trackball  to  point  at  and  click  on  its  simulated  buttons.  In  particular,  you  will  click 
the  square  buttons  in  the  center  and  the  round  buttons  on  either  side.  You  will  not  click  on  the 
labels  next  to  the  round  buttons. 


U.5.3  Logging  onto  Linescan  TnT''^**. 

Logging  on  is  the  process  of  identifying  yourself  to  Linescan  TnT'*'”  as  an  authorized  user.  Only 
authorized  users  are  allowed  to  have  access  to  the  Linescan  Tn'P”. 

Before  you  log  on 

You  need  an  operator  identification  (“ID”)  number  and  a  password  in  order  to  log  in.  You 
should  have  been  given  your  operator  ID  and  a  password  by  your  manager. 

When  you  log  on,  Linescan  TnT'''”  will  already  be  turned  on  and  the  login  screen  and  simulated 
control  panel  (figure  U-2)  will  be  displayed  on  the  left  monitor. 
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NOTE:  If  the  login  screen  and  simulated  control  panel  are  not  displayed  on  the 
left  monitor,  see  your  manager. 

Logging  on 


a.  On  the  logon  screen  and  simulated  control  panel  (figure  U-2),  enter  your  operator 
ID  number.  To  do  so,  use  the  numbered  buttons  on  the  simulated  control  panel. 

NOTE;  The  digits  1  through  9  are  on  the  square  buttons  in  the  center,  while  the 
digit  0  is  on  the  second  round  button  from  the  top  to  the  right. 

1.  Click  on  each  of  the  digits  in  the  ID  number.  (As  you  click  each  digit,  a 
beep  is  sounded.) 

2.  After  clicking  on  all  the  digits  in  the  ID,  click  the  ENTER  button. 


FIGURE  U-2.  LINESCAN  TNT™  LOGIN  SCREEN  AND  SIMULATED  CONTROL  PANEL 


b.  You  can  now  use  your  password.  To  use  your  password,  click  on  each  of  the 
digits  in  the  password,  then  click  ENTER. 
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U.5.3  Logging  Out  of  LinescanTnT™. 


Whenever  you  are  finished  using  Linescan  TnT^”  or  whenever  you  need  to  leave  it  unattended, 
even  for  a  few  minutes,  you  should  log  out  (exit).  By  logging  out,  you  ensure  that  no 
unauthorized  person  can  access  Linescan  TnT™  information. 

Logging  out 


a.  Return  to  the  Main  Menu,  if  you  are  not  already  there. 

If  starting  from  some  other  Linescan  TnT'''”  screen,  click  EXIT  as  needed  until  you  return  to  the 
Main  Menu. 


b.  At  the  Main  Menu,  click  EXIT. 

After  a  few  moments,  the  Linescan  TnT™  login  screen  and  the  simulated  control  panel  appear  on 
the  left  monitor.  Linescan  TnT™  is  ready  for  another  authorized  user  to  log  in. 

NOTE:  If  you  are  also  responsible  for  turning  off  the  Linescan  TnT™  training 
station,  see  your  manager  for  instructions. 

U.6  USING  X-RAY  MODE  (ON-LINE  TRAINING  AND  TESTING! 

This  section  tells  you  how  to  use  Linescan  TnT^”  in  conjunction  with  the  E-Scan  X-ray  machine 
for  on-line  training  and  testing. 

U.6.1  Overview  of  X-Rav  Mode. 

To  use  Linescan  TnT'''”  with  the  X-ray  machine,  you  need  to  know  about  X-ray  mode.  Threat 
Image  Projection  (TIP),  on-line  testing,  and  on-line  monitoring.  To  prepare  yourself  for  using  X- 
ray  mode  correctly,  read  the  following  overview  information  before  trying  to  use  the  X-ray  mode. 

What  is  x-ray  mode? 

When  you  use  Linescan  TnT'*'”  with  the  X-ray  machine  for  on-line  training  and  testing,  you  are 
using  X-ray  mode.  During  X-ray  mode,  you  use  the  actual  control  panel  on  the  X-ray  machine  to 
screen  parcels.  You  can  strip  the  inorganic  parts  of  an  X-ray  image,  change  it  to  reverse 
monochrome,  and  enlarge  the  image  to  get  a  closer  view. 

What  is  TIP? 

To  give  you  on-line  training  and/or  to  test  your  threat  identification  skills,  your  manager  can 
instruct  Linescan  TnT''”  to  project  threat  images  into  actual  parcels  that  pass  through  the  X-ray 
machine.  When  Linescan  TnT''”  gets  ready  to  project  a  threat  image,  it  looks  for  a  parcel  that  is 
large  enough  to  hold  the  image  it  has  to  project.  For  example,  if  the  image  scheduled  to  be 
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projected  is  a  long  item,  and  the  next  parcel  that  passes  through  the  X-ray  machine  is  a  small, 
square  package,  Linescan  TnT™  does  not  project  the  image  into  that  parcel.  It  waits  until  it  finds 
a  parcel  whose  size  is  large  enough  to  hold  the  projected  image. 

When  Linescan  TnT'*'”  does  find  a  suitable  parcel,  it  projects  the  threat  image  into  the  X-ray 
image  of  the  actual  parcel.  As  far  as  you  can  tell,  the  projected  image  looks  as  real  in  the  X-ray  as 
the  parcel  itself. 

A  manager  typically  instructs  Linescan  TnT'''”  to  project  a  certain  number  of  images  during  a 
given  period  of  time,  such  as  a  work  shift.  For  example,  he  or  she  may  schedule  20  images  to  be 
projected  over  8  hours.  Approximately  every  half  hour  an  image  will  be  projected.  (However, 
remember  that  Linescan  TnT"''”  may  occasionally  not  project  a  threat  image  if  it  cannot  find  a 
parcel  large  enough  to  hold  it.) 

What  is  on-line  training  and  testing? 

On-line  training  and  testing  is  the  mode  in  which  an  X-ray  will  be  used.  In  on-line  training  and 
testing,  you  will  follow  a  procedure  with  several  steps  or  stages. 

When  you  think  you  have  detected  a  threat  (whether  a  projected  threat  or  a  real  one),  you  will 
press  the  TIP  button  on  the  actual  X-ray  machine.  (The  TIP  button  on  the  simulated  control  panel 
of  Linescan  TnT^“  is  active  as  well.) 

At  this  time,  you  are  informed  if  you  are  being  tested  or  not. 

If  the  threat  was  a  Projected  Threat  Image,  it  is  now  erased  from  the  screen.  All  your  responses 
are  recorded  by  Linescan  TnT'''”  so  that  the  FAA  representatives  can  see  how  well  the  TIP 
function  works.  You  will  learn  more  about  this  on-line  testing  later  in  this  section. 

What  are  hits,  misses,  and  suspected  threats? 

A  hit  is  a  projected  threat  image  that  you  successfully  detected. 

A  miss  is  a  projected  threat  image  that  you  failed  to  detect.  A  miss  occurs  under  two 
circumstances: 


•  The  X-ray  into  which  the  image  was  projected  may  have  stayed  on  the  screen  for 
too  long  without  your  detecting  the  projected  threat.  The  amount  of  time  in  which 
you  have  to  respond  depends  on  how  Linescan  TnT"^”  has  been  set  up. 

•  The  X-ray  image  scrolled  (moved)  off  the  monitor  screen  before  you  pressed  the 
TIP  button. 

A  suspected  threat  is  a  threat  that  you  perceived  to  be  in  the  parcel  but  that  was  not  projected  by 
Linescan  TnT'^“.  It  might  be  a  harmless  item,  or  it  might  be  a  real  threat.  When  this  happens 
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during  on-line  monitoring,  Linescan  TnT™  automatically  saves  the  X-ray  so  that  you  and  your 
manager  can  review  it  later. 

U.6.2  Using  X-Ray  Mode. 


Starting  up  X-ray  mode. 

Before  you  can  use  X-ray  mode,  Linescan  TnT™  must  be  connected  to  the  X-ray  machine.  Your 
manager  or  a  service  technician  will  have  already  connected  them. 

a.  On  the  Main  Menu,  click  X-ray  mode. 

NOTE:  If  you  see  the  following  message  and  Linescan  TnT^“  is  not  connected  to 
the  X-ray  machine,  see  your  manager.  You  will  not  be  able  to  continue. 

PLEASE  MAKE  SURE  THE  X-RAY  MACHINE  IS  ON  AND 
ALL  CABLES  ARE  PROPERLY  CONNECTED 

b.  Now  operate  the  actual  X-ray  machine  as  you  have  been  trained  to  screen  parcels 
for  threats. 

Important !!:  The  threats  you  see  may  be  projected  threat  images 
or  they  may  be  actual  threats. 

c.  Continue  to  the  procedure  “Finding  a  threat  during  on-line  training  and  testing.” 
Finding  a  threat  during  on-line  training  and  testing. 

Recall  that  during  on-line  training  and  testing,  you  will  be  tested  with  threat  images  projected  by 
Linescan  Tn’P'“. 

Important:  Read  this  entire  procedure  before  trying  it  to  learn  about  the  actions  you  wilt  be 
required  to  take.  If  you  were  to  find  an  actual  threat  in  a  parcel,  you  must  be  able  to  act  quickly. 

d.  If  you  think  you  see  a  threat  in  a  parcel  passing  through  the  X-ray  machine,  press 
the  TIP  button  on  the  actual  control  panel. 

If  the  threat  you  see... 

•  is  a  projected  threat  image,  the  following  message  appears  on  the  monitor.  After 
reading  it,  continue  with  Step  3. 

YOU  HAVE  CORRECTLY  IDENTIFIED  A  PROJECTED  THREAT! 

CLICK  ON  THE  THREAT  OBJECT. 
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•  is  not  a  projected  threat  image,  the  following  message  appears  on  the  left  monitor: 

YOU  ARE  NOT  BEING  TESTED  AT  THIS  TIME. 

FOLLOW  THE  SECURITY  PROCEDURES  FOR  THE  SUSPECTED  THREAT! 
PRESS  DOWN  ON  THE  TRACKBALL  PAD  TO  CONTINUE. 

e.  When  you  are  ready  to  exit  X-ray  mode,  refer  to  “Exiting  X-ray  mode”  at  the  end 
of  this  section. 

U.6.3  Exiting  X-Ray  Mode. 

When  you  are  ready  to  exit  X-ray  mode  from  either  on-line  training  and  testing  or  on-line 
monitoring,  follow  this  procedure. 

a.  Press  the  TIP  button  on  the  TnT™. 

If  you  are  not  being  tested,  an  EXIT  button  appears  in  the  lower  left  corner  of  the  left  monitor  (as 
shown  in  the  example  in  figure  U-3). 

NOTE:  If  you  are  being  tested,  the  EXIT  button  does  not  appear.  You  must 
follow  the  standard  procedure  for  identifying  the  projected  threat.  Afterward,  you 
can  exit  X-ray  mode. 


VDU  Rtf  NOrSEIMBIinED  HTTIISIIHE. 

Fflumu  sEcniTv  pRoceiutfs  fur  ihe  sKncTESTiMEtrr  i 
rasn  DBUK  MTIE  raucH  no  n  continue 
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FIGURE  U-3.  EXIT  BUTTON  ON  AN  X-RAY  MODE  SCREEN 
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b.  Click  the  EXIT  button  in  the  lower  left  comer.  (If  the  button  disappears  before 
you  have  a  chance  to  click  it,  just  repeat  from  Step  1.) 

c.  X-ray  mode  is  ended.  After  a  few  moments,  you  are  returned  to  the  Main  Menu. 

SCREENER  OPERATING  CHECKLIST  FOR  TnT^M  TIP 

CARRY  OUT  THESE  STEPS  EACH  TIME  YOU  ARE  ASSIGNED 
TO  THE  X-RAY  MACHINE 

1.  _ Ensure  the  log  on  screen  is  showing  on  the  left  hand  TnT™  monitor.  If  it  is  not 

on  the  left  hand  monitor,  see  your  supervisor. 

2.  _ Enter  your  ID  number.  If  you  do  not  know  your  ID  number,  or  have  forgotten  it, 

see  your  supervisor.  Click  on  each  digit  of  your  ID  number  on  the  simulated 
control  panel  on  the  screen  using  the  trackball  and  the  clicking  pad.  After  you 
have  entered  in  all  digits,  press  ENTER. 

3.  _  Using  the  trackball  and  the  clicking  pad,  press  X-ray  Mode  on  the  Main  Menu 

4.  _  Operate  the  actual  X-ray  machine  as  you  have  been  trained  to  screen  parcels  for 

threats. 

CARRY  OUT  THESE  STEPS  EACH  TIME  YOU  LEAVE  THE  X-RAY  MACHINE 

1.  _  Press  the  TIP  button.  If  you  are  not  being  tested,  an  EXIT  button  will  appear  in 

the  lower  left  corner  of  the  monitor.  If  you  are  being  tested,  the  EXIT  button  will 
not  appear.  See  your  manager. 

2.  _ Using  the  trackball  and  the  clicking  pad,  press  the  EXIT  button  on  the  monitor 

until  you  reach  the  main  menu. 

3.  _ Using  the  trackball  and  clicking  pad,  press  EXIT  on  the  main  menu. 
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THREAT-IMAGE  PROJECTION  TRAINING  AND  TESTING 
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Appendix  W 
to  SPEARS  OT&E  TEP 


IMPROVISED  EXPLOSIVE  DEVICE  DETECTION  SYSTEM 

OT&E  LINESCAN®  TRAINING  AND  TESTING  (TnT) 
OVERVIEW  AND  INSTRUCTIONS 
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THE  LINESCAN®  TESTING  AND  TRAINING  (TnT)  OVERVIEW 


The  Linescan®  Tn'F“  system  is  a  computer-based  training  tool  developed  by  EG&G 
Astrophysics  to  help  baggage  screeners  operate  Linescan®  X-ray  equipment  with  E-Scan  and 
identify  security  threats.  The  training  system  provides  four  distinct  types  of  training  and  testing: 
training  lessons,  real-time  testing,  real-time  monitoring,  and  ongoing  training  and  testing.  The 
Operational  Test  and  Evaluation  (OT&E)  presented  baggage  screeners  with  customized  training 
lessons  on  detecting  Improvised  Explosive  Devices  (EEDs). 

The  Tn'F”  offers  on-line  and  off-line  operations.  On-line  operation  provides  real-time  training 
and  testing  where  operators  can  practice  their  skills  in  threat  identification  while  operating  an 
actual  X-ray  machine.  Off-line  training  allows  operators  to  leam  and  practice  threat 
identification  skills  by  taking  training  lessons  without  operating  the  X-ray  machine. 

Baggage  screeners  received  off-line  training  to  identify  threats  using  black/white  and  enhanced 
X-ray  equipment. 

The  Tn'n^  system  is  used  to  train  baggage  screeners  to  improve  performance  in  detecting 
threatening  objects.  Screeners  were  required  to  identify  threats  using  both  black/white  and 
enhanced  X-ray  equipment. 

Linescan®  TnT^*^  System  Components  and  Operation. 

The  Tn’F^  system  consists  of  a  combination  of  equipment  and  software.  The  equipment  consists 
of  the  following: 

a.  A  color  monitor  and  a  black-and-white  monitor  (mounted  side-by-side.  Just  as  the 
monitors  are  mounted  on  the  actual  E-Scan  X-ray  machine). 

b.  A  control  panel  (mounted  below  the  monitors)  that  replicates  the  control  panel  on 
the  actual  E-Scan  X-ray  machine. 

c.  A  trackball  (located  to  the  right  of  the  control  panel)  with  which  operators  interact 
with  the  training  programs. 

d.  A  computer  (located  inside  the  equipment  housing)  that  runs  the  training 
programs. 

e.  A  variety  of  cables  and  connectors  used  to  connect  the  training  system  to  the  X- 
ray  and  to  recording  equipment  such  as  camcorders  and  video  cassette  recorders. 

During  the  OT&E,  panicipating  baggage  screeners  received  seven  pre-determined  lessons  from 
the  Tn'F”  Lessons  menu.  All  lessons  were  previously  completed  at  the  Federal  Aviation 
Administration  (FAA)  Technical  Center  prior  to  conducting  the  OT&E.  The  FAA  selected  the 
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training  lessons  for  the  OT&E.  A  copy  of  the  Tn'P*^  operating  instructions  was  located  on  the 
TnT’’“,  and  an  FAA  representative  monitored  all  screeners  operating  the  TnT™. 

All  Linescan®  Tn'T"  lessons  were  conducted  in  an  office  behind  the  United  Airlines  group  check 
check-in  counter. 

The  following  instructions  were  provided  to  baggage  screeners  during  the  Tn’F“  training.  An 
FAA  representative  was  present  during  the  training  and  provided  baggage  screeners  with 
assistance  when  necessary. 

LINESCAN®  TRAINING  AND  TESTING  fTnTl  INSTRUCTIONS. 

The  Linescan®  TnT  training  system  is  used  to  train  baggage  screeners  to  improve  performance  in 
detecting  threat  objects.  By  taking  the  following  training  lessons,  one  will  learn  to  identify 
threats  using  both  black/white  and  enhanced  x-ray  equipment.  Review  the  seven  training  lessons 

identified  in  the  Training  Lessons  Procedures  list.  Please  do  not  take  lessons  that  are  not 

included  on  the  list. 

If  there  are  questions  at  any  time,  ask  the  Federal  Aviation  Administration  (FAA) 
representative  for  assistance. 

Getting  Started. 

If  there  are  questions  about  beginning  the  training  lessons,  turning  the  equipment  on,  or  using 
available  functions,  ask  the  FAA  representative  for  assistance. 

Use  the  following  procedures  to  begin  the  training: 

Logon  Procedures. 

1.  Using  trackball,  position  the  arrow  on  the  visual  display  number  pad. 

2.  Using  the  visual  display  number  pad,  enter  the  operator  ID:  123 

3.  Press  the  ENTER  button  to  enter  operator  ID. 

4.  Using  the  visual  display  number  pad,  enter  password:  123 

5.  Press  the  ENTER  button  to  enter  password. 

6.  Press  the  CLEAR  button  if  a  mistake  is  made  and  to  restart  the  logon  procedures. 

7.  Press  the  TIP  button  on  the  TnT  control  panel. 
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Reviewing  a  Lesson. 


After  successfully  logging  onto  the  training  system,  complete  the  following  steps: 

1.  Select  the  Lessons  button  from  the  opening  screen. 

The  Lessons  menu  appears. 

2.  On  the  Lessons  menu,  click  Review. 

The  Review  menu  appears. 

3.  To  review  a  prior  lesson,  click  A  Prior  Lesson. 

A  checklist  of  the  lessons  previously  completed  is  displayed.  It  shows  up 
to  10  lessons  at  a  time.  To  bring  other  lessons  into  view,  use  the  up  and 
down  arrows  to  the  right  of  the  lessons. 

4.  To  select  the  lesson  to  review,  click  on  it. 

Table  W-1  shows  a  list  of  the  lessons  to  review.  Review  the  lessons  in  the 
order  presented  in  table  W-l. 

TABLE  W-1.  LIST  OF  LESSONS  TO  REVIEW 


Unit  2 

Lesson  1 

Unit  4 

Lesson  1 

Unit  4 

Lesson  3 

Units 

Lesson  4 

Units 

Lesson  1 

Unit  6 

Lesson  3 

Unit? 

Lesson  1 

When  the  lesson  is  presented,  complete  it  by  closely  reading  the 
information  presented  on  the  screens. 

5.  Begin  the  appropriate  lesson  displayed  on  the  right  visual  display  (Lesson  Screen). 

6.  When  finished  reviewing  a  lesson,  immediately  review  the  corresponding  test. 
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Appendix  X 
to  SPEARS  OT&E  TEP 


TEST  BAG  CONnGURATIONS 


Requests  for  descriptions  of  test  bag  configurations  should  be  submitted  in  writing 
to  the  FAA  program  manager  supporting  this  effort. 
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Appendix  Y 
to  SPEARS  OT&ETEP 


CHECKED-CARRY-ON  BAGGAGE  -  THREAT  ARTICLE  DEFINITION  SHEETS 
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Requests  for  descriptions  of  threat  article  definitions  should  be  submitted  in 
writing  to  the  FAA  program  manager  supporting  this  effort. 
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Appendix  Z 
to  SPEARS  OT&E  TEP 


THREAT-IMAGE  PROJECTION  INSERTION  PROTOCOL 
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TIP  1  INSERTION  PROTOCOL 
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TIP  2  INSERTION  PROTOCOL 
WEEKS  4-6 
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to  SPEARS  OT&ETEP 


NORTHWEST  AIRLINES  DAILY  DEPARTURES  -  LOS  ANGELES 
INTERNATIONAL  AIRPORT 
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NORTHWEST  AIRLINES  DAILY  DEPARTURES 
LOS  ANGELES  INTERNATIONAL  AIRPORT 


TIME 

DEPARTURES 

#/Total 

6:30 

MSP  RSW 

6:50 

FAT 

0.07 

7:10 

MEM  SAN 

7:30 

SEA 

7:50 

MSP  FAT 

0.07 

8:10 

BOS  PSP 

8:30 

DTW  SAN  YYZ 

8:50 

HNL  NGO  SBP 

0.11 

9:10 

MRY  PSP 

9:30 

SBA 

9:50 

SAN  SEA 

0,04 

10:10 

10:30 

PHX 

10:50 

LAS 

0.03 

11:10 

11:30 

BOS  FAT  SAN  SAN 

11:50 

LAS  MSP  SFO 

0.15 

12:10 

DTW  HNL  KIX  MNL 

12:30 

LGW  FAT  PHX  SAN  SEA 

12:50 

MRY  ONT  PSP  SBA  SBP 

0.20 

13:10 

MEM  MKE  NRT  PEK  SHA 

13:30 

13:50 

SEL 

0.01 

14:10 

14:30 

BOS  MSP 

14:50 

SAN 

0.04 

15:10 

15:30 

FAT  PSP  SAN 

15:50 

MSP 

0.09 

16:10 

AMS  DTW  MRY 

16:30 

16:50 

17:10 

17:30 

17:50 

SAN 

0.01 

18:10 

18:30 

FAT  ONT 

18:50 

0.03 

19:10 

19:30 

LAS 

19:50 

PSP 

0.04 

20:10 

SAN 

20:30 

20:50 

FAT 

0.04 

21:10 

MRY  SBA 

21:30 

21:50 

22:10 

22:30 

22:50 

DTW  FAT  LAS 

0.08 

23:10 

ONT  PSP  SBP 

Leaend: 

AMS 

Amsterdam 

BOS 

Boston 

DTW 

Detroit 

FAT 

Fresno 

HNL 

Honolulu 

KIX  , 

Osaka 

LAS 

Las  Vegas 

LGA 

LaGuardia 

LGW 

Gatwick 

MEM 

Memphis 

MIA 

Miami 

MKE 

Milwaukee 

MNL 

Manila 

MRY 

Monterey 

MSP 

Minneapolis 

NGO 

Nagaro 

NRT 

Narita 

ONT 

Ontario 

PEK 

Beejing 

PHX 

Pheonix 

PSP 

Palm  Springs 

RSW 

Fort  Meyers 

SAN 

San  Diego 

SBA 

Santa  Barbara 

SBP 

San  Luis  Obispo 

SEA 

Seattle 

SEL 

Seoul 

SFO 

San  Francisco 

SHA 

Shangai 

YYZ 

Toronto 

Note:  Northwest  Airlines 

accounts  for  approximately 

75%  of  enplanements 

using  this  checkpoint. 

Other  airlines  using  this 

terminal  checkpoint  include 

Virgin  Air,  Aslana,  Air 

Canada,  Hawaiian  Air, 

Vast,  and  Air  New  Zealand. 
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to  SPEARS  OT&E  TEP 


SPEARS  OT&E  EG&G  ASTROPHYSICS  LINESCAN 
TESTING  AND  TRAINING  SYSTEM  (TNT™)  USABILITY  SURVEY 
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SPEARS  OT&E  EG&G  ASTROPHYSICS  LINESCAN 
TESTING  AND  TRAINING  SYSTEM  (TNT™)  USABILITY  SURVEY 

SUBJECT  NUMBER: _  DATE: _ 

The  FA  A  wants  to  know  your  opinion  of  the  TnT™  training  and  testing  you  just  received.  Your 
ratings  will  play  an  important  role  in  evaluating  the  TnT™  system.  All  information  will  be  kept 
confidential,  so  please  express  your  honest  opinion.  Your  participation  is  greatly  appreciated. 

If  you  have  any  questions  while  completing  this  survey,  please  ask  the  FAA 
representative  for  assistance. 

Please  circle  the  number  that  best  indicates  your  opinion  of  the  statements  in  the  following 
survey. 
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Not  at  all 
1 


2 


3 


Very  much  so 
5 


1.  The  SPEARS  TIP  training  and  testing  improved  1  2  3  4  5 

my  ability  to  detect  explosive  devices. 

2.  The  SPEARS  TIP  training  and  testing  will  help  1  2  3  4  5 

me  on  my  job. 

3.  I  enjoyed  taking  the  SPEARS  TIP  training  and  1  2  3  4  5 

testing. 

4.  I  would  like  to  continue  this  training  in  my  1  2  3  4  5 

work. 

5.  The  SPEARS  TIP  function  is  easy  to  operate.  1  2  3  4  5 

6.  The  TIP  key  is  easy  to  locate  and  depress.  1  2  3  4  5 

7.  The  SPEARS  machine  showed  me  explosive  1  2  3  4  5 

images  I  had  never  seen  before. 

8.  The  false  explosive  images  look  like  real  x-ray  1  2  3  4  5 

images  of  explosive  devices. 

9.  I  could  not  guess  when  I  would  get  a  false  1  2  3  4  5 

explosive  image  from  the  SPEARS. 

10.  The  SPEARS  generated  reports  were  1  2  3  4  5 

understandable. 

11.  The  SPEARS  machine  helped  me  to  1  2  3  4  5 

understand  when  I  made  the  right  and  wrong 

decision  about  whether  there  was  an  explosive 
device  in  a  bag. 

12.  The  SPEARS  machine  did  not  slow  down  the  1  2  3  4  5 

movement  of  baggage  and  passengers. 

13.  The  SPEARS  machine  did  not  interfere  with  1  2  3  4  5 

the  performance  of  my  tasks. 

14.  The  SPEARS  machine  is  conveniently  located.  1  2  3  4  5 

15.  The  SPEARS  machine  x-ray  enhancement  1  2  3  4  5 

buttons  improved  my  ability  to  detect  explosive 

devices. 
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INSERTION  REPRESENTATIVENESS  ASSESSMENT 
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INSERTION  REPRESENTATIVENESS  ASSESSMENT 
ABBREVIATED  DATA  COLLECTION  FORM 


Instructions:  You  will  be  presented  individual  images  of  baggage,  which  may  or  may  not  contain 
one  or  more  threat  objects.  You  will  rate  how  “real”  each  of  these  images  is,  based  on  the 
likelihood  of  their  appearance  during  your  regular  working  day. 

When  examining  each  image  for  “realism”  and  the  likelihood  of  encountering  it,  please  consider 
the  following: 


a.  Types  of  objects  in  image. 

b.  Density  of  objects  in  image. 

c.  Placement  of  objects  in  image. 

d.  Bag  type. 

Please  rate  each  image  using  the  scale  from  1  to  7  in  the  following,  indicating  how  likely  it 
would  be  to  find  this  image  in  the  real  world.  Put  an  “X”  in  the  space  that  matches  your  choice. 


Would  Never 
Encounter 


Very  Likely 
to  Encounter 


Image  Number 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

1 

2 

3 

• 

. 

, 

• 

99 

100 
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SPEARS  OT&E  FTI  THREAT  ITEM  LOCATION  MATRIX 
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SPEARS  OT&E  rn  THREAT  ITEM  LOCATOR  MATRIX 


DATE: _ 

IMAGE  NUMBER: 


1 

2 

3 

4 

5 

6 

7 

00 

9 

This  matrix  will  be  used  to  track  the  location  of  projected  fictional  threat  images  within  x-ray 
images. 
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to  SPEARS  OT&ETEP 


SUBJECT  CHECBCLIST  FICTIONAL  IMAGE  PROJECTION  (HP)  TRAINING  AND 
TESTING  IMPROVISED  EXPLOSIVE  DEVICE  (lED)  DETECTION  TEST 
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SUBJECT  CHECKLIST 
TIP  TRAINING  AND  TESTING 
lED  DETECTION  TEST 


PILOT  TEST 


IIESSSSI 

Password 

Date 

PI 

PI 

P2 

P2 

This  checklist  will  be  used  to  record  the  dates  of  fictional  image  projection  training  and 
improvised  explosive  device  detecting  testing. 
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